Note: This is test shard 5 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from LocationAwareRebalancingBasicTest
[ RUN ] LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250627 13:52:13.796456 413 test_util.cc:276] Using random seed: -62979015
I20250627 13:52:13.808369 413 ts_itest-base.cc:115] Starting cluster with:
I20250627 13:52:13.808455 413 ts_itest-base.cc:116] --------------
I20250627 13:52:13.808516 413 ts_itest-base.cc:117] 6 tablet servers
I20250627 13:52:13.808584 413 ts_itest-base.cc:118] 3 replicas per TS
I20250627 13:52:13.808660 413 ts_itest-base.cc:119] --------------
2025-06-27T13:52:13Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-27T13:52:13Z Disabled control of system clock
I20250627 13:52:13.833563 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.103.126:44433
--webserver_interface=127.0.103.126
--webserver_port=0
--builtin_ntp_servers=127.0.103.84:39581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.103.126:44433
--location_mapping_cmd=/tmp/dist-test-taskNzQG6V/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20250627 13:52:14.157429 425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:14.157833 425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:52:14.157953 425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:14.167521 425 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:52:14.167665 425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250627 13:52:14.167780 425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:52:14.167845 425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250627 13:52:14.167927 425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250627 13:52:14.178764 425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:39581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskNzQG6V/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--ipki_ca_key_size=768
--master_addresses=127.0.103.126:44433
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.103.126:44433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.103.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:52:14.180374 425 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:52:14.182135 425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:52:14.192790 430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:14.192917 433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:14.192763 431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:52:14.193363 425 server_base.cc:1048] running on GCE node
I20250627 13:52:14.194180 425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:52:14.195570 425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:52:14.196791 425 hybrid_clock.cc:648] HybridClock initialized: now 1751032334196684 us; error 78 us; skew 500 ppm
I20250627 13:52:14.197079 425 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:52:14.203713 425 webserver.cc:469] Webserver started at http://127.0.103.126:43691/ using document root <none> and password file <none>
I20250627 13:52:14.204466 425 fs_manager.cc:362] Metadata directory not provided
I20250627 13:52:14.204593 425 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:52:14.204979 425 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:52:14.207912 425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3deba1b1926b493aa67245d8111cb55c"
format_stamp: "Formatted at 2025-06-27 13:52:14 on dist-test-slave-1ckg"
I20250627 13:52:14.208601 425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3deba1b1926b493aa67245d8111cb55c"
format_stamp: "Formatted at 2025-06-27 13:52:14 on dist-test-slave-1ckg"
I20250627 13:52:14.214036 425 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20250627 13:52:14.217149 439 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:14.218436 425 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250627 13:52:14.218623 425 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3deba1b1926b493aa67245d8111cb55c"
format_stamp: "Formatted at 2025-06-27 13:52:14 on dist-test-slave-1ckg"
I20250627 13:52:14.218791 425 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:52:14.238652 425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:52:14.239462 425 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:52:14.239744 425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:52:14.260013 425 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:44433
I20250627 13:52:14.260035 490 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:44433 every 8 connection(s)
I20250627 13:52:14.261915 425 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250627 13:52:14.266387 491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:14.270529 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 425
I20250627 13:52:14.270780 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250627 13:52:14.278817 491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3deba1b1926b493aa67245d8111cb55c: Bootstrap starting.
I20250627 13:52:14.282508 491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3deba1b1926b493aa67245d8111cb55c: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:14.283601 491 log.cc:826] T 00000000000000000000000000000000 P 3deba1b1926b493aa67245d8111cb55c: Log is configured to *not* fsync() on all Append() calls
I20250627 13:52:14.286157 491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3deba1b1926b493aa67245d8111cb55c: No bootstrap required, opened a new log
I20250627 13:52:14.292333 491 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3deba1b1926b493aa67245d8111cb55c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3deba1b1926b493aa67245d8111cb55c" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 44433 } }
I20250627 13:52:14.292793 491 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3deba1b1926b493aa67245d8111cb55c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:14.292919 491 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3deba1b1926b493aa67245d8111cb55c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3deba1b1926b493aa67245d8111cb55c, State: Initialized, Role: FOLLOWER
I20250627 13:52:14.293594 491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3deba1b1926b493aa67245d8111cb55c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3deba1b1926b493aa67245d8111cb55c" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 44433 } }
I20250627 13:52:14.293874 491 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3deba1b1926b493aa67245d8111cb55c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:52:14.293980 491 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3deba1b1926b493aa67245d8111cb55c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:52:14.294181 491 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3deba1b1926b493aa67245d8111cb55c [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:14.296643 491 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3deba1b1926b493aa67245d8111cb55c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3deba1b1926b493aa67245d8111cb55c" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 44433 } }
I20250627 13:52:14.297210 491 leader_election.cc:304] T 00000000000000000000000000000000 P 3deba1b1926b493aa67245d8111cb55c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3deba1b1926b493aa67245d8111cb55c; no voters:
I20250627 13:52:14.297900 491 leader_election.cc:290] T 00000000000000000000000000000000 P 3deba1b1926b493aa67245d8111cb55c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250627 13:52:14.297994 496 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3deba1b1926b493aa67245d8111cb55c [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:14.300017 491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3deba1b1926b493aa67245d8111cb55c [sys.catalog]: configured and running, proceeding with master startup.
I20250627 13:52:14.300693 496 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3deba1b1926b493aa67245d8111cb55c [term 1 LEADER]: Becoming Leader. State: Replica: 3deba1b1926b493aa67245d8111cb55c, State: Running, Role: LEADER
I20250627 13:52:14.301349 496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3deba1b1926b493aa67245d8111cb55c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3deba1b1926b493aa67245d8111cb55c" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 44433 } }
I20250627 13:52:14.304591 498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3deba1b1926b493aa67245d8111cb55c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3deba1b1926b493aa67245d8111cb55c. Latest consensus state: current_term: 1 leader_uuid: "3deba1b1926b493aa67245d8111cb55c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3deba1b1926b493aa67245d8111cb55c" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 44433 } } }
I20250627 13:52:14.304857 498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3deba1b1926b493aa67245d8111cb55c [sys.catalog]: This master's current role is: LEADER
I20250627 13:52:14.305330 497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3deba1b1926b493aa67245d8111cb55c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3deba1b1926b493aa67245d8111cb55c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3deba1b1926b493aa67245d8111cb55c" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 44433 } } }
I20250627 13:52:14.305619 497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3deba1b1926b493aa67245d8111cb55c [sys.catalog]: This master's current role is: LEADER
I20250627 13:52:14.310948 505 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250627 13:52:14.317103 505 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250627 13:52:14.333185 505 catalog_manager.cc:1349] Generated new cluster ID: 602629e837394fd49d829a8e5d2b1d77
I20250627 13:52:14.333315 505 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250627 13:52:14.358322 505 catalog_manager.cc:1372] Generated new certificate authority record
I20250627 13:52:14.359500 505 catalog_manager.cc:1506] Loading token signing keys...
I20250627 13:52:14.367277 505 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 3deba1b1926b493aa67245d8111cb55c: Generated new TSK 0
I20250627 13:52:14.368247 505 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250627 13:52:14.382386 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.65:0
--local_ip_for_outbound_sockets=127.0.103.65
--webserver_interface=127.0.103.65
--webserver_port=0
--tserver_master_addrs=127.0.103.126:44433
--builtin_ntp_servers=127.0.103.84:39581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250627 13:52:14.719372 516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:14.719762 516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:52:14.719884 516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:14.731359 516 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:52:14.731524 516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:52:14.731753 516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.65
I20250627 13:52:14.745532 516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:39581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.103.65
--webserver_port=0
--tserver_master_addrs=127.0.103.126:44433
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.65
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:52:14.747170 516 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:52:14.748952 516 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:52:14.760955 521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:14.761065 524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:14.760979 522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:52:14.762017 516 server_base.cc:1048] running on GCE node
I20250627 13:52:14.762604 516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:52:14.763492 516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:52:14.764709 516 hybrid_clock.cc:648] HybridClock initialized: now 1751032334764633 us; error 56 us; skew 500 ppm
I20250627 13:52:14.764986 516 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:52:14.767652 516 webserver.cc:469] Webserver started at http://127.0.103.65:36249/ using document root <none> and password file <none>
I20250627 13:52:14.768323 516 fs_manager.cc:362] Metadata directory not provided
I20250627 13:52:14.768440 516 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:52:14.768743 516 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:52:14.771080 516 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "9f767210a91f44ae8b83b994131eee98"
format_stamp: "Formatted at 2025-06-27 13:52:14 on dist-test-slave-1ckg"
I20250627 13:52:14.771790 516 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "9f767210a91f44ae8b83b994131eee98"
format_stamp: "Formatted at 2025-06-27 13:52:14 on dist-test-slave-1ckg"
I20250627 13:52:14.776661 516 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20250627 13:52:14.779893 530 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:14.781090 516 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20250627 13:52:14.781239 516 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9f767210a91f44ae8b83b994131eee98"
format_stamp: "Formatted at 2025-06-27 13:52:14 on dist-test-slave-1ckg"
I20250627 13:52:14.781404 516 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:52:14.802019 516 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:52:14.802819 516 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:52:14.803071 516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:52:14.803913 516 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:52:14.805608 516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:52:14.805717 516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250627 13:52:14.805847 516 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:52:14.805907 516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:14.840930 516 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:43023
I20250627 13:52:14.840972 642 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:43023 every 8 connection(s)
I20250627 13:52:14.842342 516 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250627 13:52:14.851245 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 516
I20250627 13:52:14.851491 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250627 13:52:14.855360 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.66:0
--local_ip_for_outbound_sockets=127.0.103.66
--webserver_interface=127.0.103.66
--webserver_port=0
--tserver_master_addrs=127.0.103.126:44433
--builtin_ntp_servers=127.0.103.84:39581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:52:14.856320 643 heartbeater.cc:344] Connected to a master server at 127.0.103.126:44433
I20250627 13:52:14.857398 643 heartbeater.cc:461] Registering TS with master...
I20250627 13:52:14.858474 643 heartbeater.cc:507] Master 127.0.103.126:44433 requested a full tablet report, sending...
I20250627 13:52:14.903992 456 ts_manager.cc:194] Registered new tserver with Master: 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023)
I20250627 13:52:14.906497 456 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.65:58509
W20250627 13:52:15.184223 647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:15.184556 647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:52:15.184681 647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:15.192677 647 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:52:15.192804 647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:52:15.192996 647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.66
I20250627 13:52:15.202479 647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:39581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.103.66
--webserver_port=0
--tserver_master_addrs=127.0.103.126:44433
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.66
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:52:15.204010 647 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:52:15.205799 647 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:52:15.217197 653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:15.217303 654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:15.217274 656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:52:15.218127 647 server_base.cc:1048] running on GCE node
I20250627 13:52:15.218729 647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:52:15.219616 647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:52:15.220844 647 hybrid_clock.cc:648] HybridClock initialized: now 1751032335220776 us; error 45 us; skew 500 ppm
I20250627 13:52:15.221132 647 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:52:15.223372 647 webserver.cc:469] Webserver started at http://127.0.103.66:39257/ using document root <none> and password file <none>
I20250627 13:52:15.224082 647 fs_manager.cc:362] Metadata directory not provided
I20250627 13:52:15.224203 647 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:52:15.224514 647 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:52:15.230965 647 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "75c4f59060974a3b823305b9dd339a7e"
format_stamp: "Formatted at 2025-06-27 13:52:15 on dist-test-slave-1ckg"
I20250627 13:52:15.231720 647 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "75c4f59060974a3b823305b9dd339a7e"
format_stamp: "Formatted at 2025-06-27 13:52:15 on dist-test-slave-1ckg"
I20250627 13:52:15.237121 647 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20250627 13:52:15.240465 662 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:15.241780 647 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250627 13:52:15.241933 647 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "75c4f59060974a3b823305b9dd339a7e"
format_stamp: "Formatted at 2025-06-27 13:52:15 on dist-test-slave-1ckg"
I20250627 13:52:15.242094 647 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:52:15.283108 647 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:52:15.283946 647 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:52:15.284210 647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:52:15.285085 647 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:52:15.286845 647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:52:15.286939 647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:15.287068 647 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:52:15.287127 647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:15.322574 647 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:36499
I20250627 13:52:15.322636 774 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:36499 every 8 connection(s)
I20250627 13:52:15.324028 647 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250627 13:52:15.333742 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 647
I20250627 13:52:15.333894 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250627 13:52:15.337402 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.67:0
--local_ip_for_outbound_sockets=127.0.103.67
--webserver_interface=127.0.103.67
--webserver_port=0
--tserver_master_addrs=127.0.103.126:44433
--builtin_ntp_servers=127.0.103.84:39581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:52:15.337976 775 heartbeater.cc:344] Connected to a master server at 127.0.103.126:44433
I20250627 13:52:15.338353 775 heartbeater.cc:461] Registering TS with master...
I20250627 13:52:15.339210 775 heartbeater.cc:507] Master 127.0.103.126:44433 requested a full tablet report, sending...
I20250627 13:52:15.397572 456 ts_manager.cc:194] Registered new tserver with Master: 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499)
I20250627 13:52:15.398597 456 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.66:45611
W20250627 13:52:15.682034 778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:15.682356 778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:52:15.682488 778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:15.689716 778 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:52:15.689864 778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:52:15.690033 778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.67
I20250627 13:52:15.699323 778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:39581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.103.67
--webserver_port=0
--tserver_master_addrs=127.0.103.126:44433
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.67
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:52:15.700907 778 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:52:15.702777 778 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:52:15.714390 784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:15.714432 785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:15.714460 787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:52:15.715214 778 server_base.cc:1048] running on GCE node
I20250627 13:52:15.715770 778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:52:15.716671 778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:52:15.717873 778 hybrid_clock.cc:648] HybridClock initialized: now 1751032335717817 us; error 44 us; skew 500 ppm
I20250627 13:52:15.718138 778 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:52:15.720261 778 webserver.cc:469] Webserver started at http://127.0.103.67:37645/ using document root <none> and password file <none>
I20250627 13:52:15.720928 778 fs_manager.cc:362] Metadata directory not provided
I20250627 13:52:15.721046 778 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:52:15.721366 778 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:52:15.723693 778 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "fa66786862da49ae826fb519c7d89b5e"
format_stamp: "Formatted at 2025-06-27 13:52:15 on dist-test-slave-1ckg"
I20250627 13:52:15.724500 778 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "fa66786862da49ae826fb519c7d89b5e"
format_stamp: "Formatted at 2025-06-27 13:52:15 on dist-test-slave-1ckg"
I20250627 13:52:15.729423 778 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20250627 13:52:15.732558 793 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:15.734419 778 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20250627 13:52:15.734560 778 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "fa66786862da49ae826fb519c7d89b5e"
format_stamp: "Formatted at 2025-06-27 13:52:15 on dist-test-slave-1ckg"
I20250627 13:52:15.734731 778 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:52:15.755952 778 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:52:15.756757 778 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:52:15.757007 778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:52:15.758016 778 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:52:15.759477 778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:52:15.759555 778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:15.759685 778 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:52:15.759739 778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:15.793910 778 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:40747
I20250627 13:52:15.793951 905 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:40747 every 8 connection(s)
I20250627 13:52:15.795205 778 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250627 13:52:15.805573 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 778
I20250627 13:52:15.805774 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250627 13:52:15.808056 906 heartbeater.cc:344] Connected to a master server at 127.0.103.126:44433
I20250627 13:52:15.808396 906 heartbeater.cc:461] Registering TS with master...
I20250627 13:52:15.808871 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.68:0
--local_ip_for_outbound_sockets=127.0.103.68
--webserver_interface=127.0.103.68
--webserver_port=0
--tserver_master_addrs=127.0.103.126:44433
--builtin_ntp_servers=127.0.103.84:39581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:52:15.809226 906 heartbeater.cc:507] Master 127.0.103.126:44433 requested a full tablet report, sending...
I20250627 13:52:15.854508 456 ts_manager.cc:194] Registered new tserver with Master: fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747)
I20250627 13:52:15.855926 456 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.67:43223
I20250627 13:52:15.910246 643 heartbeater.cc:499] Master 127.0.103.126:44433 was elected leader, sending a full tablet report...
W20250627 13:52:16.129727 909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:16.130036 909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:52:16.130195 909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:16.138325 909 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:52:16.138429 909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:52:16.138594 909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.68
I20250627 13:52:16.147557 909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:39581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.103.68
--webserver_port=0
--tserver_master_addrs=127.0.103.126:44433
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.68
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:52:16.149098 909 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:52:16.150880 909 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:52:16.164265 915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:16.164232 916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:16.164289 918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:52:16.165017 909 server_base.cc:1048] running on GCE node
I20250627 13:52:16.165496 909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:52:16.166396 909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:52:16.167599 909 hybrid_clock.cc:648] HybridClock initialized: now 1751032336167548 us; error 26 us; skew 500 ppm
I20250627 13:52:16.167879 909 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:52:16.170040 909 webserver.cc:469] Webserver started at http://127.0.103.68:43045/ using document root <none> and password file <none>
I20250627 13:52:16.170786 909 fs_manager.cc:362] Metadata directory not provided
I20250627 13:52:16.170900 909 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:52:16.171727 909 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:52:16.173997 909 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "50e8c97891d74b38b5c130d3bcbabcb6"
format_stamp: "Formatted at 2025-06-27 13:52:16 on dist-test-slave-1ckg"
I20250627 13:52:16.174701 909 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "50e8c97891d74b38b5c130d3bcbabcb6"
format_stamp: "Formatted at 2025-06-27 13:52:16 on dist-test-slave-1ckg"
I20250627 13:52:16.179715 909 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250627 13:52:16.182710 924 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:16.183934 909 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250627 13:52:16.184065 909 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "50e8c97891d74b38b5c130d3bcbabcb6"
format_stamp: "Formatted at 2025-06-27 13:52:16 on dist-test-slave-1ckg"
I20250627 13:52:16.184226 909 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:52:16.216665 909 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:52:16.217484 909 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:52:16.217787 909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:52:16.218726 909 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:52:16.220186 909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:52:16.220253 909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:16.220371 909 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:52:16.220415 909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:16.254801 909 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:34289
I20250627 13:52:16.254837 1036 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:34289 every 8 connection(s)
I20250627 13:52:16.256203 909 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250627 13:52:16.257225 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 909
I20250627 13:52:16.257381 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250627 13:52:16.261215 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.69:0
--local_ip_for_outbound_sockets=127.0.103.69
--webserver_interface=127.0.103.69
--webserver_port=0
--tserver_master_addrs=127.0.103.126:44433
--builtin_ntp_servers=127.0.103.84:39581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:52:16.269713 1037 heartbeater.cc:344] Connected to a master server at 127.0.103.126:44433
I20250627 13:52:16.270054 1037 heartbeater.cc:461] Registering TS with master...
I20250627 13:52:16.270824 1037 heartbeater.cc:507] Master 127.0.103.126:44433 requested a full tablet report, sending...
I20250627 13:52:16.316561 456 ts_manager.cc:194] Registered new tserver with Master: 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68:34289)
I20250627 13:52:16.317480 456 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.68:47491
I20250627 13:52:16.401548 775 heartbeater.cc:499] Master 127.0.103.126:44433 was elected leader, sending a full tablet report...
W20250627 13:52:16.591758 1040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:16.592103 1040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:52:16.592226 1040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:16.600311 1040 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:52:16.600441 1040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:52:16.600625 1040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.69
I20250627 13:52:16.610536 1040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:39581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.103.69
--webserver_port=0
--tserver_master_addrs=127.0.103.126:44433
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.69
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:52:16.612213 1040 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:52:16.614058 1040 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:52:16.625962 1049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:16.625998 1046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:16.625913 1047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:52:16.627082 1040 server_base.cc:1048] running on GCE node
I20250627 13:52:16.627661 1040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:52:16.628573 1040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:52:16.629781 1040 hybrid_clock.cc:648] HybridClock initialized: now 1751032336629710 us; error 48 us; skew 500 ppm
I20250627 13:52:16.630072 1040 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:52:16.632272 1040 webserver.cc:469] Webserver started at http://127.0.103.69:37507/ using document root <none> and password file <none>
I20250627 13:52:16.632961 1040 fs_manager.cc:362] Metadata directory not provided
I20250627 13:52:16.633105 1040 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:52:16.633430 1040 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:52:16.635921 1040 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "957a8a8be3f542f48e8824e581fd0f89"
format_stamp: "Formatted at 2025-06-27 13:52:16 on dist-test-slave-1ckg"
I20250627 13:52:16.636657 1040 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "957a8a8be3f542f48e8824e581fd0f89"
format_stamp: "Formatted at 2025-06-27 13:52:16 on dist-test-slave-1ckg"
I20250627 13:52:16.641582 1040 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20250627 13:52:16.644690 1055 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:16.645936 1040 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250627 13:52:16.646107 1040 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "957a8a8be3f542f48e8824e581fd0f89"
format_stamp: "Formatted at 2025-06-27 13:52:16 on dist-test-slave-1ckg"
I20250627 13:52:16.646265 1040 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:52:16.669770 1040 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:52:16.670676 1040 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:52:16.670965 1040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:52:16.671947 1040 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:52:16.673470 1040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:52:16.673548 1040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:16.673697 1040 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:52:16.673763 1040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250627 13:52:16.709373 1040 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.69:42325
I20250627 13:52:16.710006 1167 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.69:42325 every 8 connection(s)
I20250627 13:52:16.710947 1040 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250627 13:52:16.712347 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 1040
I20250627 13:52:16.712512 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250627 13:52:16.715556 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.70:0
--local_ip_for_outbound_sockets=127.0.103.70
--webserver_interface=127.0.103.70
--webserver_port=0
--tserver_master_addrs=127.0.103.126:44433
--builtin_ntp_servers=127.0.103.84:39581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:52:16.731005 1168 heartbeater.cc:344] Connected to a master server at 127.0.103.126:44433
I20250627 13:52:16.731350 1168 heartbeater.cc:461] Registering TS with master...
I20250627 13:52:16.732168 1168 heartbeater.cc:507] Master 127.0.103.126:44433 requested a full tablet report, sending...
I20250627 13:52:16.792755 456 ts_manager.cc:194] Registered new tserver with Master: 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69:42325)
I20250627 13:52:16.794080 456 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.69:39013
I20250627 13:52:16.858846 906 heartbeater.cc:499] Master 127.0.103.126:44433 was elected leader, sending a full tablet report...
W20250627 13:52:17.071420 1171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:17.071739 1171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:52:17.071835 1171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:17.079927 1171 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:52:17.080041 1171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:52:17.080166 1171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.70
I20250627 13:52:17.090066 1171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:39581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.103.70
--webserver_port=0
--tserver_master_addrs=127.0.103.126:44433
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.70
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:52:17.091733 1171 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:52:17.093525 1171 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:52:17.105764 1178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:17.106495 1177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:17.105835 1180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:52:17.107224 1171 server_base.cc:1048] running on GCE node
I20250627 13:52:17.107848 1171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:52:17.108780 1171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:52:17.110008 1171 hybrid_clock.cc:648] HybridClock initialized: now 1751032337109928 us; error 59 us; skew 500 ppm
I20250627 13:52:17.110304 1171 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:52:17.112691 1171 webserver.cc:469] Webserver started at http://127.0.103.70:45913/ using document root <none> and password file <none>
I20250627 13:52:17.113440 1171 fs_manager.cc:362] Metadata directory not provided
I20250627 13:52:17.113572 1171 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:52:17.113960 1171 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:52:17.116488 1171 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "5c28ff2fe9424379819d478fdd093cfc"
format_stamp: "Formatted at 2025-06-27 13:52:17 on dist-test-slave-1ckg"
I20250627 13:52:17.117300 1171 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "5c28ff2fe9424379819d478fdd093cfc"
format_stamp: "Formatted at 2025-06-27 13:52:17 on dist-test-slave-1ckg"
I20250627 13:52:17.122582 1171 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20250627 13:52:17.125941 1186 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:17.127245 1171 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250627 13:52:17.127411 1171 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "5c28ff2fe9424379819d478fdd093cfc"
format_stamp: "Formatted at 2025-06-27 13:52:17 on dist-test-slave-1ckg"
I20250627 13:52:17.127581 1171 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:52:17.145646 1171 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:52:17.146648 1171 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:52:17.146929 1171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:52:17.147846 1171 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:52:17.149363 1171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:52:17.149446 1171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:17.149573 1171 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:52:17.149629 1171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:17.185386 1171 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.70:39887
I20250627 13:52:17.185423 1298 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.70:39887 every 8 connection(s)
I20250627 13:52:17.186901 1171 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20250627 13:52:17.196341 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 1171
I20250627 13:52:17.196478 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20250627 13:52:17.200861 1299 heartbeater.cc:344] Connected to a master server at 127.0.103.126:44433
I20250627 13:52:17.201264 1299 heartbeater.cc:461] Registering TS with master...
I20250627 13:52:17.202116 1299 heartbeater.cc:507] Master 127.0.103.126:44433 requested a full tablet report, sending...
I20250627 13:52:17.249697 455 ts_manager.cc:194] Registered new tserver with Master: 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70:39887)
I20250627 13:52:17.250969 455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.70:44927
I20250627 13:52:17.252291 413 external_mini_cluster.cc:934] 6 TS(s) registered with all masters
I20250627 13:52:17.274842 413 external_mini_cluster.cc:934] 6 TS(s) registered with all masters
I20250627 13:52:17.320888 1037 heartbeater.cc:499] Master 127.0.103.126:44433 was elected leader, sending a full tablet report...
I20250627 13:52:17.797217 1168 heartbeater.cc:499] Master 127.0.103.126:44433 was elected leader, sending a full tablet report...
I20250627 13:52:18.254153 1299 heartbeater.cc:499] Master 127.0.103.126:44433 was elected leader, sending a full tablet report...
I20250627 13:52:21.029074 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49852:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250627 13:52:21.103655 1234 tablet_service.cc:1468] Processing CreateTablet for tablet acf517032e5744da93a91d4348b6ceed (DEFAULT_TABLE table=rebalance_test_table_0 [id=d8ff1c89a2454d51bf27ea32a020b242]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:21.105865 1102 tablet_service.cc:1468] Processing CreateTablet for tablet 578c441034d447f99fa557b63e643fba (DEFAULT_TABLE table=rebalance_test_table_0 [id=d8ff1c89a2454d51bf27ea32a020b242]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:21.107300 1234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acf517032e5744da93a91d4348b6ceed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:21.107298 1102 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578c441034d447f99fa557b63e643fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:21.107612 1103 tablet_service.cc:1468] Processing CreateTablet for tablet a4be1509d80e400c9a1cd0686f38e47e (DEFAULT_TABLE table=rebalance_test_table_0 [id=d8ff1c89a2454d51bf27ea32a020b242]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:21.108105 1103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4be1509d80e400c9a1cd0686f38e47e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:21.108592 841 tablet_service.cc:1468] Processing CreateTablet for tablet acf517032e5744da93a91d4348b6ceed (DEFAULT_TABLE table=rebalance_test_table_0 [id=d8ff1c89a2454d51bf27ea32a020b242]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:21.108780 972 tablet_service.cc:1468] Processing CreateTablet for tablet a4be1509d80e400c9a1cd0686f38e47e (DEFAULT_TABLE table=rebalance_test_table_0 [id=d8ff1c89a2454d51bf27ea32a020b242]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:21.110060 972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4be1509d80e400c9a1cd0686f38e47e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:21.110049 840 tablet_service.cc:1468] Processing CreateTablet for tablet 578c441034d447f99fa557b63e643fba (DEFAULT_TABLE table=rebalance_test_table_0 [id=d8ff1c89a2454d51bf27ea32a020b242]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:21.111112 841 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acf517032e5744da93a91d4348b6ceed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:21.117489 840 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578c441034d447f99fa557b63e643fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:21.134485 1326 tablet_bootstrap.cc:492] T 578c441034d447f99fa557b63e643fba P fa66786862da49ae826fb519c7d89b5e: Bootstrap starting.
I20250627 13:52:21.139617 578 tablet_service.cc:1468] Processing CreateTablet for tablet a4be1509d80e400c9a1cd0686f38e47e (DEFAULT_TABLE table=rebalance_test_table_0 [id=d8ff1c89a2454d51bf27ea32a020b242]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:21.139958 577 tablet_service.cc:1468] Processing CreateTablet for tablet 578c441034d447f99fa557b63e643fba (DEFAULT_TABLE table=rebalance_test_table_0 [id=d8ff1c89a2454d51bf27ea32a020b242]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:21.140498 1325 tablet_bootstrap.cc:492] T a4be1509d80e400c9a1cd0686f38e47e P 50e8c97891d74b38b5c130d3bcbabcb6: Bootstrap starting.
I20250627 13:52:21.141163 1326 tablet_bootstrap.cc:654] T 578c441034d447f99fa557b63e643fba P fa66786862da49ae826fb519c7d89b5e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:21.141278 577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578c441034d447f99fa557b63e643fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:21.142491 1326 log.cc:826] T 578c441034d447f99fa557b63e643fba P fa66786862da49ae826fb519c7d89b5e: Log is configured to *not* fsync() on all Append() calls
I20250627 13:52:21.143741 578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4be1509d80e400c9a1cd0686f38e47e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:21.144948 1325 tablet_bootstrap.cc:654] T a4be1509d80e400c9a1cd0686f38e47e P 50e8c97891d74b38b5c130d3bcbabcb6: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:21.146970 1325 log.cc:826] T a4be1509d80e400c9a1cd0686f38e47e P 50e8c97891d74b38b5c130d3bcbabcb6: Log is configured to *not* fsync() on all Append() calls
I20250627 13:52:21.146847 710 tablet_service.cc:1468] Processing CreateTablet for tablet acf517032e5744da93a91d4348b6ceed (DEFAULT_TABLE table=rebalance_test_table_0 [id=d8ff1c89a2454d51bf27ea32a020b242]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:21.148505 710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acf517032e5744da93a91d4348b6ceed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:21.156239 1325 tablet_bootstrap.cc:492] T a4be1509d80e400c9a1cd0686f38e47e P 50e8c97891d74b38b5c130d3bcbabcb6: No bootstrap required, opened a new log
I20250627 13:52:21.156620 1325 ts_tablet_manager.cc:1397] T a4be1509d80e400c9a1cd0686f38e47e P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.005s
I20250627 13:52:21.157893 1324 tablet_bootstrap.cc:492] T a4be1509d80e400c9a1cd0686f38e47e P 957a8a8be3f542f48e8824e581fd0f89: Bootstrap starting.
I20250627 13:52:21.164055 1325 raft_consensus.cc:357] T a4be1509d80e400c9a1cd0686f38e47e P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:21.164642 1325 raft_consensus.cc:383] T a4be1509d80e400c9a1cd0686f38e47e P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:21.164808 1325 raft_consensus.cc:738] T a4be1509d80e400c9a1cd0686f38e47e P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e8c97891d74b38b5c130d3bcbabcb6, State: Initialized, Role: FOLLOWER
I20250627 13:52:21.165813 1324 tablet_bootstrap.cc:654] T a4be1509d80e400c9a1cd0686f38e47e P 957a8a8be3f542f48e8824e581fd0f89: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:21.165638 1325 consensus_queue.cc:260] T a4be1509d80e400c9a1cd0686f38e47e P 50e8c97891d74b38b5c130d3bcbabcb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:21.171311 1327 tablet_bootstrap.cc:492] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc: Bootstrap starting.
I20250627 13:52:21.171679 1324 log.cc:826] T a4be1509d80e400c9a1cd0686f38e47e P 957a8a8be3f542f48e8824e581fd0f89: Log is configured to *not* fsync() on all Append() calls
I20250627 13:52:21.174801 1325 ts_tablet_manager.cc:1428] T a4be1509d80e400c9a1cd0686f38e47e P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent starting tablet: real 0.018s user 0.007s sys 0.009s
I20250627 13:52:21.175338 1327 tablet_bootstrap.cc:654] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:21.175817 1331 tablet_bootstrap.cc:492] T a4be1509d80e400c9a1cd0686f38e47e P 9f767210a91f44ae8b83b994131eee98: Bootstrap starting.
I20250627 13:52:21.177845 1326 tablet_bootstrap.cc:492] T 578c441034d447f99fa557b63e643fba P fa66786862da49ae826fb519c7d89b5e: No bootstrap required, opened a new log
I20250627 13:52:21.178210 1326 ts_tablet_manager.cc:1397] T 578c441034d447f99fa557b63e643fba P fa66786862da49ae826fb519c7d89b5e: Time spent bootstrapping tablet: real 0.044s user 0.008s sys 0.000s
I20250627 13:52:21.184947 1326 raft_consensus.cc:357] T 578c441034d447f99fa557b63e643fba P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:21.186162 1331 tablet_bootstrap.cc:654] T a4be1509d80e400c9a1cd0686f38e47e P 9f767210a91f44ae8b83b994131eee98: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:21.188022 1331 log.cc:826] T a4be1509d80e400c9a1cd0686f38e47e P 9f767210a91f44ae8b83b994131eee98: Log is configured to *not* fsync() on all Append() calls
I20250627 13:52:21.188216 1327 log.cc:826] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc: Log is configured to *not* fsync() on all Append() calls
I20250627 13:52:21.188844 1326 raft_consensus.cc:383] T 578c441034d447f99fa557b63e643fba P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:21.189034 1326 raft_consensus.cc:738] T 578c441034d447f99fa557b63e643fba P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa66786862da49ae826fb519c7d89b5e, State: Initialized, Role: FOLLOWER
I20250627 13:52:21.189878 1326 consensus_queue.cc:260] T 578c441034d447f99fa557b63e643fba P fa66786862da49ae826fb519c7d89b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:21.191515 1324 tablet_bootstrap.cc:492] T a4be1509d80e400c9a1cd0686f38e47e P 957a8a8be3f542f48e8824e581fd0f89: No bootstrap required, opened a new log
I20250627 13:52:21.191852 1324 ts_tablet_manager.cc:1397] T a4be1509d80e400c9a1cd0686f38e47e P 957a8a8be3f542f48e8824e581fd0f89: Time spent bootstrapping tablet: real 0.052s user 0.015s sys 0.007s
I20250627 13:52:21.196777 1326 ts_tablet_manager.cc:1428] T 578c441034d447f99fa557b63e643fba P fa66786862da49ae826fb519c7d89b5e: Time spent starting tablet: real 0.018s user 0.011s sys 0.000s
I20250627 13:52:21.198459 1333 tablet_bootstrap.cc:492] T acf517032e5744da93a91d4348b6ceed P 75c4f59060974a3b823305b9dd339a7e: Bootstrap starting.
I20250627 13:52:21.200474 1324 raft_consensus.cc:357] T a4be1509d80e400c9a1cd0686f38e47e P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:21.200837 1324 raft_consensus.cc:383] T a4be1509d80e400c9a1cd0686f38e47e P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:21.201928 1324 raft_consensus.cc:738] T a4be1509d80e400c9a1cd0686f38e47e P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957a8a8be3f542f48e8824e581fd0f89, State: Initialized, Role: FOLLOWER
I20250627 13:52:21.202859 1324 consensus_queue.cc:260] T a4be1509d80e400c9a1cd0686f38e47e P 957a8a8be3f542f48e8824e581fd0f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:21.204540 1333 tablet_bootstrap.cc:654] T acf517032e5744da93a91d4348b6ceed P 75c4f59060974a3b823305b9dd339a7e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:21.204860 1326 tablet_bootstrap.cc:492] T acf517032e5744da93a91d4348b6ceed P fa66786862da49ae826fb519c7d89b5e: Bootstrap starting.
I20250627 13:52:21.205960 1333 log.cc:826] T acf517032e5744da93a91d4348b6ceed P 75c4f59060974a3b823305b9dd339a7e: Log is configured to *not* fsync() on all Append() calls
I20250627 13:52:21.208456 1327 tablet_bootstrap.cc:492] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc: No bootstrap required, opened a new log
I20250627 13:52:21.208896 1327 ts_tablet_manager.cc:1397] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.008s
I20250627 13:52:21.213864 1326 tablet_bootstrap.cc:654] T acf517032e5744da93a91d4348b6ceed P fa66786862da49ae826fb519c7d89b5e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:21.215029 1331 tablet_bootstrap.cc:492] T a4be1509d80e400c9a1cd0686f38e47e P 9f767210a91f44ae8b83b994131eee98: No bootstrap required, opened a new log
I20250627 13:52:21.215355 1331 ts_tablet_manager.cc:1397] T a4be1509d80e400c9a1cd0686f38e47e P 9f767210a91f44ae8b83b994131eee98: Time spent bootstrapping tablet: real 0.040s user 0.020s sys 0.000s
I20250627 13:52:21.215638 1333 tablet_bootstrap.cc:492] T acf517032e5744da93a91d4348b6ceed P 75c4f59060974a3b823305b9dd339a7e: No bootstrap required, opened a new log
I20250627 13:52:21.215922 1333 ts_tablet_manager.cc:1397] T acf517032e5744da93a91d4348b6ceed P 75c4f59060974a3b823305b9dd339a7e: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.010s
I20250627 13:52:21.222633 1324 ts_tablet_manager.cc:1428] T a4be1509d80e400c9a1cd0686f38e47e P 957a8a8be3f542f48e8824e581fd0f89: Time spent starting tablet: real 0.031s user 0.020s sys 0.008s
I20250627 13:52:21.222523 1333 raft_consensus.cc:357] T acf517032e5744da93a91d4348b6ceed P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:21.223098 1333 raft_consensus.cc:383] T acf517032e5744da93a91d4348b6ceed P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:21.223165 1324 tablet_bootstrap.cc:492] T 578c441034d447f99fa557b63e643fba P 957a8a8be3f542f48e8824e581fd0f89: Bootstrap starting.
I20250627 13:52:21.223821 1333 raft_consensus.cc:738] T acf517032e5744da93a91d4348b6ceed P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c4f59060974a3b823305b9dd339a7e, State: Initialized, Role: FOLLOWER
I20250627 13:52:21.224670 1333 consensus_queue.cc:260] T acf517032e5744da93a91d4348b6ceed P 75c4f59060974a3b823305b9dd339a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:21.224628 1327 raft_consensus.cc:357] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:21.224989 1327 raft_consensus.cc:383] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:21.225057 1324 tablet_bootstrap.cc:654] T 578c441034d447f99fa557b63e643fba P 957a8a8be3f542f48e8824e581fd0f89: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:21.225121 1327 raft_consensus.cc:738] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c28ff2fe9424379819d478fdd093cfc, State: Initialized, Role: FOLLOWER
I20250627 13:52:21.227933 1327 consensus_queue.cc:260] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:21.229933 1331 raft_consensus.cc:357] T a4be1509d80e400c9a1cd0686f38e47e P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:21.231462 1326 tablet_bootstrap.cc:492] T acf517032e5744da93a91d4348b6ceed P fa66786862da49ae826fb519c7d89b5e: No bootstrap required, opened a new log
I20250627 13:52:21.231716 1326 ts_tablet_manager.cc:1397] T acf517032e5744da93a91d4348b6ceed P fa66786862da49ae826fb519c7d89b5e: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.007s
I20250627 13:52:21.232683 1326 raft_consensus.cc:357] T acf517032e5744da93a91d4348b6ceed P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:21.232939 1326 raft_consensus.cc:383] T acf517032e5744da93a91d4348b6ceed P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:21.233057 1326 raft_consensus.cc:738] T acf517032e5744da93a91d4348b6ceed P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa66786862da49ae826fb519c7d89b5e, State: Initialized, Role: FOLLOWER
I20250627 13:52:21.233121 1333 ts_tablet_manager.cc:1428] T acf517032e5744da93a91d4348b6ceed P 75c4f59060974a3b823305b9dd339a7e: Time spent starting tablet: real 0.017s user 0.008s sys 0.007s
I20250627 13:52:21.233354 1326 consensus_queue.cc:260] T acf517032e5744da93a91d4348b6ceed P fa66786862da49ae826fb519c7d89b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:21.235280 1327 ts_tablet_manager.cc:1428] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc: Time spent starting tablet: real 0.026s user 0.010s sys 0.005s
I20250627 13:52:21.235996 1326 ts_tablet_manager.cc:1428] T acf517032e5744da93a91d4348b6ceed P fa66786862da49ae826fb519c7d89b5e: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250627 13:52:21.236361 1331 raft_consensus.cc:383] T a4be1509d80e400c9a1cd0686f38e47e P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:21.236601 1331 raft_consensus.cc:738] T a4be1509d80e400c9a1cd0686f38e47e P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f767210a91f44ae8b83b994131eee98, State: Initialized, Role: FOLLOWER
I20250627 13:52:21.237253 1342 raft_consensus.cc:491] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:21.237548 1342 raft_consensus.cc:513] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:21.237622 1331 consensus_queue.cc:260] T a4be1509d80e400c9a1cd0686f38e47e P 9f767210a91f44ae8b83b994131eee98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:21.241529 1324 tablet_bootstrap.cc:492] T 578c441034d447f99fa557b63e643fba P 957a8a8be3f542f48e8824e581fd0f89: No bootstrap required, opened a new log
I20250627 13:52:21.241817 1324 ts_tablet_manager.cc:1397] T 578c441034d447f99fa557b63e643fba P 957a8a8be3f542f48e8824e581fd0f89: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.008s
I20250627 13:52:21.243081 1324 raft_consensus.cc:357] T 578c441034d447f99fa557b63e643fba P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:21.243422 1324 raft_consensus.cc:383] T 578c441034d447f99fa557b63e643fba P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:21.243553 1324 raft_consensus.cc:738] T 578c441034d447f99fa557b63e643fba P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957a8a8be3f542f48e8824e581fd0f89, State: Initialized, Role: FOLLOWER
I20250627 13:52:21.243894 1324 consensus_queue.cc:260] T 578c441034d447f99fa557b63e643fba P 957a8a8be3f542f48e8824e581fd0f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:21.244197 1342 leader_election.cc:290] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747), 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499)
I20250627 13:52:21.244719 1324 ts_tablet_manager.cc:1428] T 578c441034d447f99fa557b63e643fba P 957a8a8be3f542f48e8824e581fd0f89: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250627 13:52:21.246606 1331 ts_tablet_manager.cc:1428] T a4be1509d80e400c9a1cd0686f38e47e P 9f767210a91f44ae8b83b994131eee98: Time spent starting tablet: real 0.031s user 0.008s sys 0.008s
I20250627 13:52:21.247133 1331 tablet_bootstrap.cc:492] T 578c441034d447f99fa557b63e643fba P 9f767210a91f44ae8b83b994131eee98: Bootstrap starting.
I20250627 13:52:21.249107 1337 raft_consensus.cc:491] T 578c441034d447f99fa557b63e643fba P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:21.249351 1337 raft_consensus.cc:513] T 578c441034d447f99fa557b63e643fba P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:21.249678 1331 tablet_bootstrap.cc:654] T 578c441034d447f99fa557b63e643fba P 9f767210a91f44ae8b83b994131eee98: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:21.253415 861 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "acf517032e5744da93a91d4348b6ceed" candidate_uuid: "5c28ff2fe9424379819d478fdd093cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e" is_pre_election: true
I20250627 13:52:21.253693 1337 leader_election.cc:290] T 578c441034d447f99fa557b63e643fba P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023), fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747)
I20250627 13:52:21.253937 861 raft_consensus.cc:2466] T acf517032e5744da93a91d4348b6ceed P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c28ff2fe9424379819d478fdd093cfc in term 0.
I20250627 13:52:21.254618 1189 leader_election.cc:304] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c28ff2fe9424379819d478fdd093cfc, fa66786862da49ae826fb519c7d89b5e; no voters:
I20250627 13:52:21.256065 730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "acf517032e5744da93a91d4348b6ceed" candidate_uuid: "5c28ff2fe9424379819d478fdd093cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c4f59060974a3b823305b9dd339a7e" is_pre_election: true
I20250627 13:52:21.256609 730 raft_consensus.cc:2466] T acf517032e5744da93a91d4348b6ceed P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c28ff2fe9424379819d478fdd093cfc in term 0.
I20250627 13:52:21.256821 1342 raft_consensus.cc:2802] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:21.257019 1342 raft_consensus.cc:491] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:21.257119 1342 raft_consensus.cc:3058] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:21.259724 1331 tablet_bootstrap.cc:492] T 578c441034d447f99fa557b63e643fba P 9f767210a91f44ae8b83b994131eee98: No bootstrap required, opened a new log
I20250627 13:52:21.259930 1331 ts_tablet_manager.cc:1397] T 578c441034d447f99fa557b63e643fba P 9f767210a91f44ae8b83b994131eee98: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.011s
W20250627 13:52:21.260923 1038 tablet.cc:2378] T a4be1509d80e400c9a1cd0686f38e47e P 50e8c97891d74b38b5c130d3bcbabcb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:52:21.260893 1342 raft_consensus.cc:513] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:21.261946 1331 raft_consensus.cc:357] T 578c441034d447f99fa557b63e643fba P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:21.262163 1331 raft_consensus.cc:383] T 578c441034d447f99fa557b63e643fba P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:21.262105 861 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "acf517032e5744da93a91d4348b6ceed" candidate_uuid: "5c28ff2fe9424379819d478fdd093cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e"
I20250627 13:52:21.262269 1331 raft_consensus.cc:738] T 578c441034d447f99fa557b63e643fba P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f767210a91f44ae8b83b994131eee98, State: Initialized, Role: FOLLOWER
I20250627 13:52:21.262315 861 raft_consensus.cc:3058] T acf517032e5744da93a91d4348b6ceed P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:21.262528 1331 consensus_queue.cc:260] T 578c441034d447f99fa557b63e643fba P 9f767210a91f44ae8b83b994131eee98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:21.263036 1331 ts_tablet_manager.cc:1428] T 578c441034d447f99fa557b63e643fba P 9f767210a91f44ae8b83b994131eee98: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250627 13:52:21.265396 861 raft_consensus.cc:2466] T acf517032e5744da93a91d4348b6ceed P fa66786862da49ae826fb519c7d89b5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c28ff2fe9424379819d478fdd093cfc in term 1.
I20250627 13:52:21.265921 1189 leader_election.cc:304] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c28ff2fe9424379819d478fdd093cfc, fa66786862da49ae826fb519c7d89b5e; no voters:
I20250627 13:52:21.270328 730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "acf517032e5744da93a91d4348b6ceed" candidate_uuid: "5c28ff2fe9424379819d478fdd093cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c4f59060974a3b823305b9dd339a7e"
I20250627 13:52:21.270592 730 raft_consensus.cc:3058] T acf517032e5744da93a91d4348b6ceed P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:21.274755 730 raft_consensus.cc:2466] T acf517032e5744da93a91d4348b6ceed P 75c4f59060974a3b823305b9dd339a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c28ff2fe9424379819d478fdd093cfc in term 1.
I20250627 13:52:21.275665 1342 leader_election.cc:290] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc [CANDIDATE]: Term 1 election: Requested vote from peers fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747), 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499)
I20250627 13:52:21.276084 1342 raft_consensus.cc:2802] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:21.276741 1342 raft_consensus.cc:695] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc [term 1 LEADER]: Becoming Leader. State: Replica: 5c28ff2fe9424379819d478fdd093cfc, State: Running, Role: LEADER
I20250627 13:52:21.277585 1342 consensus_queue.cc:237] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:21.281577 861 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "578c441034d447f99fa557b63e643fba" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e" is_pre_election: true
I20250627 13:52:21.282146 861 raft_consensus.cc:2466] T 578c441034d447f99fa557b63e643fba P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 0.
I20250627 13:52:21.283108 1058 leader_election.cc:304] T 578c441034d447f99fa557b63e643fba P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, fa66786862da49ae826fb519c7d89b5e; no voters:
I20250627 13:52:21.283689 1337 raft_consensus.cc:2802] T 578c441034d447f99fa557b63e643fba P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:21.283847 1337 raft_consensus.cc:491] T 578c441034d447f99fa557b63e643fba P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:21.283942 1337 raft_consensus.cc:3058] T 578c441034d447f99fa557b63e643fba P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:21.287755 1337 raft_consensus.cc:513] T 578c441034d447f99fa557b63e643fba P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:21.289085 1337 leader_election.cc:290] T 578c441034d447f99fa557b63e643fba P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 election: Requested vote from peers 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023), fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747)
I20250627 13:52:21.288552 455 catalog_manager.cc:5582] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70). New cstate: current_term: 1 leader_uuid: "5c28ff2fe9424379819d478fdd093cfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } health_report { overall_health: UNKNOWN } } }
I20250627 13:52:21.289412 597 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "578c441034d447f99fa557b63e643fba" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98"
I20250627 13:52:21.289430 598 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "578c441034d447f99fa557b63e643fba" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98" is_pre_election: true
I20250627 13:52:21.290032 597 raft_consensus.cc:3058] T 578c441034d447f99fa557b63e643fba P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:21.290282 861 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "578c441034d447f99fa557b63e643fba" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e"
I20250627 13:52:21.290488 861 raft_consensus.cc:3058] T 578c441034d447f99fa557b63e643fba P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:21.292164 1337 raft_consensus.cc:491] T a4be1509d80e400c9a1cd0686f38e47e P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:21.292325 1337 raft_consensus.cc:513] T a4be1509d80e400c9a1cd0686f38e47e P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:21.293061 1337 leader_election.cc:290] T a4be1509d80e400c9a1cd0686f38e47e P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023), 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68:34289)
I20250627 13:52:21.293357 861 raft_consensus.cc:2466] T 578c441034d447f99fa557b63e643fba P fa66786862da49ae826fb519c7d89b5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 1.
I20250627 13:52:21.293401 597 raft_consensus.cc:2466] T 578c441034d447f99fa557b63e643fba P 9f767210a91f44ae8b83b994131eee98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 1.
I20250627 13:52:21.293582 598 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4be1509d80e400c9a1cd0686f38e47e" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98" is_pre_election: true
I20250627 13:52:21.293818 598 raft_consensus.cc:2466] T a4be1509d80e400c9a1cd0686f38e47e P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 0.
I20250627 13:52:21.294198 1058 leader_election.cc:304] T 578c441034d447f99fa557b63e643fba P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, fa66786862da49ae826fb519c7d89b5e; no voters:
I20250627 13:52:21.294504 1337 raft_consensus.cc:2802] T 578c441034d447f99fa557b63e643fba P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:21.294509 1056 leader_election.cc:304] T a4be1509d80e400c9a1cd0686f38e47e P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, 9f767210a91f44ae8b83b994131eee98; no voters:
I20250627 13:52:21.295073 1337 raft_consensus.cc:695] T 578c441034d447f99fa557b63e643fba P 957a8a8be3f542f48e8824e581fd0f89 [term 1 LEADER]: Becoming Leader. State: Replica: 957a8a8be3f542f48e8824e581fd0f89, State: Running, Role: LEADER
I20250627 13:52:21.295307 1351 raft_consensus.cc:2802] T a4be1509d80e400c9a1cd0686f38e47e P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:21.295431 1351 raft_consensus.cc:491] T a4be1509d80e400c9a1cd0686f38e47e P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:21.295588 1351 raft_consensus.cc:3058] T a4be1509d80e400c9a1cd0686f38e47e P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:21.295619 1337 consensus_queue.cc:237] T 578c441034d447f99fa557b63e643fba P 957a8a8be3f542f48e8824e581fd0f89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:21.299840 1351 raft_consensus.cc:513] T a4be1509d80e400c9a1cd0686f38e47e P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
W20250627 13:52:21.300524 907 tablet.cc:2378] T 578c441034d447f99fa557b63e643fba P fa66786862da49ae826fb519c7d89b5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:52:21.300791 1351 leader_election.cc:290] T a4be1509d80e400c9a1cd0686f38e47e P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 election: Requested vote from peers 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023), 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68:34289)
I20250627 13:52:21.302820 992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4be1509d80e400c9a1cd0686f38e47e" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" is_pre_election: true
I20250627 13:52:21.303334 992 raft_consensus.cc:2466] T a4be1509d80e400c9a1cd0686f38e47e P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 0.
I20250627 13:52:21.303910 598 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4be1509d80e400c9a1cd0686f38e47e" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98"
I20250627 13:52:21.304154 598 raft_consensus.cc:3058] T a4be1509d80e400c9a1cd0686f38e47e P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:21.303704 991 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4be1509d80e400c9a1cd0686f38e47e" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e8c97891d74b38b5c130d3bcbabcb6"
I20250627 13:52:21.304435 991 raft_consensus.cc:3058] T a4be1509d80e400c9a1cd0686f38e47e P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:21.304243 455 catalog_manager.cc:5582] T 578c441034d447f99fa557b63e643fba P 957a8a8be3f542f48e8824e581fd0f89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69). New cstate: current_term: 1 leader_uuid: "957a8a8be3f542f48e8824e581fd0f89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } health_report { overall_health: HEALTHY } } }
I20250627 13:52:21.306846 598 raft_consensus.cc:2466] T a4be1509d80e400c9a1cd0686f38e47e P 9f767210a91f44ae8b83b994131eee98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 1.
I20250627 13:52:21.307026 991 raft_consensus.cc:2466] T a4be1509d80e400c9a1cd0686f38e47e P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 1.
I20250627 13:52:21.307350 1056 leader_election.cc:304] T a4be1509d80e400c9a1cd0686f38e47e P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, 9f767210a91f44ae8b83b994131eee98; no voters:
I20250627 13:52:21.307613 1352 raft_consensus.cc:2802] T a4be1509d80e400c9a1cd0686f38e47e P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:21.307749 1352 raft_consensus.cc:695] T a4be1509d80e400c9a1cd0686f38e47e P 957a8a8be3f542f48e8824e581fd0f89 [term 1 LEADER]: Becoming Leader. State: Replica: 957a8a8be3f542f48e8824e581fd0f89, State: Running, Role: LEADER
I20250627 13:52:21.308028 1352 consensus_queue.cc:237] T a4be1509d80e400c9a1cd0686f38e47e P 957a8a8be3f542f48e8824e581fd0f89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:21.310540 455 catalog_manager.cc:5582] T a4be1509d80e400c9a1cd0686f38e47e P 957a8a8be3f542f48e8824e581fd0f89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69). New cstate: current_term: 1 leader_uuid: "957a8a8be3f542f48e8824e581fd0f89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } health_report { overall_health: HEALTHY } } }
W20250627 13:52:21.329567 776 tablet.cc:2378] T acf517032e5744da93a91d4348b6ceed P 75c4f59060974a3b823305b9dd339a7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250627 13:52:21.348762 644 tablet.cc:2378] T 578c441034d447f99fa557b63e643fba P 9f767210a91f44ae8b83b994131eee98: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250627 13:52:21.441406 1300 tablet.cc:2378] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250627 13:52:21.466048 1169 tablet.cc:2378] T 578c441034d447f99fa557b63e643fba P 957a8a8be3f542f48e8824e581fd0f89: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250627 13:52:21.647347 1355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:21.648124 1355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:21.655553 1355 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20250627 13:52:21.747872 1342 consensus_queue.cc:1035] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:21.766952 1352 consensus_queue.cc:1035] T a4be1509d80e400c9a1cd0686f38e47e P 957a8a8be3f542f48e8824e581fd0f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:21.779464 1349 consensus_queue.cc:1035] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:21.781476 1352 consensus_queue.cc:1035] T 578c441034d447f99fa557b63e643fba P 957a8a8be3f542f48e8824e581fd0f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:21.856344 1352 consensus_queue.cc:1035] T 578c441034d447f99fa557b63e643fba P 957a8a8be3f542f48e8824e581fd0f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:21.894048 1352 consensus_queue.cc:1035] T a4be1509d80e400c9a1cd0686f38e47e P 957a8a8be3f542f48e8824e581fd0f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 367.88 ms
time per row: 0.18394 ms
I20250627 13:52:22.188292 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49852:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250627 13:52:22.198073 1234 tablet_service.cc:1468] Processing CreateTablet for tablet 445539c2574245009333771c3464fcb9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6b4b7982bf14328ba009b600d28f833]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:22.198559 1234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 445539c2574245009333771c3464fcb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:22.198577 710 tablet_service.cc:1468] Processing CreateTablet for tablet 445539c2574245009333771c3464fcb9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6b4b7982bf14328ba009b600d28f833]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:22.198949 710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 445539c2574245009333771c3464fcb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:22.199776 840 tablet_service.cc:1468] Processing CreateTablet for tablet 46cbed417f5a4f2ebded566383ee3440 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6b4b7982bf14328ba009b600d28f833]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:22.200150 840 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46cbed417f5a4f2ebded566383ee3440. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:22.202085 578 tablet_service.cc:1468] Processing CreateTablet for tablet 46cbed417f5a4f2ebded566383ee3440 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6b4b7982bf14328ba009b600d28f833]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:22.202541 578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46cbed417f5a4f2ebded566383ee3440. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:22.203471 972 tablet_service.cc:1468] Processing CreateTablet for tablet 445539c2574245009333771c3464fcb9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6b4b7982bf14328ba009b600d28f833]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:22.203872 972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 445539c2574245009333771c3464fcb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:22.204685 1103 tablet_service.cc:1468] Processing CreateTablet for tablet 46cbed417f5a4f2ebded566383ee3440 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6b4b7982bf14328ba009b600d28f833]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:22.205083 1103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46cbed417f5a4f2ebded566383ee3440. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:22.205720 709 tablet_service.cc:1468] Processing CreateTablet for tablet b7e50b9da893466486b668fcd0aac962 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6b4b7982bf14328ba009b600d28f833]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:22.206192 709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7e50b9da893466486b668fcd0aac962. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:22.206179 971 tablet_service.cc:1468] Processing CreateTablet for tablet b7e50b9da893466486b668fcd0aac962 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6b4b7982bf14328ba009b600d28f833]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:22.206588 971 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7e50b9da893466486b668fcd0aac962. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:22.210152 1233 tablet_service.cc:1468] Processing CreateTablet for tablet b7e50b9da893466486b668fcd0aac962 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d6b4b7982bf14328ba009b600d28f833]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:22.210651 1233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7e50b9da893466486b668fcd0aac962. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:22.226475 1429 tablet_bootstrap.cc:492] T 46cbed417f5a4f2ebded566383ee3440 P 9f767210a91f44ae8b83b994131eee98: Bootstrap starting.
I20250627 13:52:22.229131 1426 tablet_bootstrap.cc:492] T 46cbed417f5a4f2ebded566383ee3440 P 957a8a8be3f542f48e8824e581fd0f89: Bootstrap starting.
I20250627 13:52:22.230656 1430 tablet_bootstrap.cc:492] T 445539c2574245009333771c3464fcb9 P 5c28ff2fe9424379819d478fdd093cfc: Bootstrap starting.
I20250627 13:52:22.232288 1428 tablet_bootstrap.cc:492] T 445539c2574245009333771c3464fcb9 P 50e8c97891d74b38b5c130d3bcbabcb6: Bootstrap starting.
I20250627 13:52:22.233359 1430 tablet_bootstrap.cc:654] T 445539c2574245009333771c3464fcb9 P 5c28ff2fe9424379819d478fdd093cfc: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:22.234959 1426 tablet_bootstrap.cc:654] T 46cbed417f5a4f2ebded566383ee3440 P 957a8a8be3f542f48e8824e581fd0f89: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:22.234952 1428 tablet_bootstrap.cc:654] T 445539c2574245009333771c3464fcb9 P 50e8c97891d74b38b5c130d3bcbabcb6: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:22.236644 1427 tablet_bootstrap.cc:492] T 46cbed417f5a4f2ebded566383ee3440 P fa66786862da49ae826fb519c7d89b5e: Bootstrap starting.
I20250627 13:52:22.237972 1425 tablet_bootstrap.cc:492] T 445539c2574245009333771c3464fcb9 P 75c4f59060974a3b823305b9dd339a7e: Bootstrap starting.
I20250627 13:52:22.239406 1430 tablet_bootstrap.cc:492] T 445539c2574245009333771c3464fcb9 P 5c28ff2fe9424379819d478fdd093cfc: No bootstrap required, opened a new log
I20250627 13:52:22.239563 1430 ts_tablet_manager.cc:1397] T 445539c2574245009333771c3464fcb9 P 5c28ff2fe9424379819d478fdd093cfc: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20250627 13:52:22.239804 1429 tablet_bootstrap.cc:654] T 46cbed417f5a4f2ebded566383ee3440 P 9f767210a91f44ae8b83b994131eee98: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:22.240295 1425 tablet_bootstrap.cc:654] T 445539c2574245009333771c3464fcb9 P 75c4f59060974a3b823305b9dd339a7e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:22.240607 1430 raft_consensus.cc:357] T 445539c2574245009333771c3464fcb9 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:22.240821 1430 raft_consensus.cc:383] T 445539c2574245009333771c3464fcb9 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:22.240913 1430 raft_consensus.cc:738] T 445539c2574245009333771c3464fcb9 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c28ff2fe9424379819d478fdd093cfc, State: Initialized, Role: FOLLOWER
I20250627 13:52:22.241142 1430 consensus_queue.cc:260] T 445539c2574245009333771c3464fcb9 P 5c28ff2fe9424379819d478fdd093cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:22.241865 1430 ts_tablet_manager.cc:1428] T 445539c2574245009333771c3464fcb9 P 5c28ff2fe9424379819d478fdd093cfc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250627 13:52:22.242237 1430 tablet_bootstrap.cc:492] T b7e50b9da893466486b668fcd0aac962 P 5c28ff2fe9424379819d478fdd093cfc: Bootstrap starting.
I20250627 13:52:22.244251 1427 tablet_bootstrap.cc:654] T 46cbed417f5a4f2ebded566383ee3440 P fa66786862da49ae826fb519c7d89b5e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:22.244454 1430 tablet_bootstrap.cc:654] T b7e50b9da893466486b668fcd0aac962 P 5c28ff2fe9424379819d478fdd093cfc: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:22.247305 1428 tablet_bootstrap.cc:492] T 445539c2574245009333771c3464fcb9 P 50e8c97891d74b38b5c130d3bcbabcb6: No bootstrap required, opened a new log
I20250627 13:52:22.247965 1428 ts_tablet_manager.cc:1397] T 445539c2574245009333771c3464fcb9 P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20250627 13:52:22.248739 1426 tablet_bootstrap.cc:492] T 46cbed417f5a4f2ebded566383ee3440 P 957a8a8be3f542f48e8824e581fd0f89: No bootstrap required, opened a new log
I20250627 13:52:22.249138 1428 raft_consensus.cc:357] T 445539c2574245009333771c3464fcb9 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:22.249349 1428 raft_consensus.cc:383] T 445539c2574245009333771c3464fcb9 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:22.249385 1425 tablet_bootstrap.cc:492] T 445539c2574245009333771c3464fcb9 P 75c4f59060974a3b823305b9dd339a7e: No bootstrap required, opened a new log
I20250627 13:52:22.249436 1428 raft_consensus.cc:738] T 445539c2574245009333771c3464fcb9 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e8c97891d74b38b5c130d3bcbabcb6, State: Initialized, Role: FOLLOWER
I20250627 13:52:22.249569 1425 ts_tablet_manager.cc:1397] T 445539c2574245009333771c3464fcb9 P 75c4f59060974a3b823305b9dd339a7e: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20250627 13:52:22.249717 1428 consensus_queue.cc:260] T 445539c2574245009333771c3464fcb9 P 50e8c97891d74b38b5c130d3bcbabcb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:22.250468 1426 ts_tablet_manager.cc:1397] T 46cbed417f5a4f2ebded566383ee3440 P 957a8a8be3f542f48e8824e581fd0f89: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.005s
I20250627 13:52:22.250836 1428 ts_tablet_manager.cc:1428] T 445539c2574245009333771c3464fcb9 P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250627 13:52:22.251145 1428 tablet_bootstrap.cc:492] T b7e50b9da893466486b668fcd0aac962 P 50e8c97891d74b38b5c130d3bcbabcb6: Bootstrap starting.
I20250627 13:52:22.251302 1425 raft_consensus.cc:357] T 445539c2574245009333771c3464fcb9 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:22.251654 1425 raft_consensus.cc:383] T 445539c2574245009333771c3464fcb9 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:22.251809 1425 raft_consensus.cc:738] T 445539c2574245009333771c3464fcb9 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c4f59060974a3b823305b9dd339a7e, State: Initialized, Role: FOLLOWER
I20250627 13:52:22.252180 1425 consensus_queue.cc:260] T 445539c2574245009333771c3464fcb9 P 75c4f59060974a3b823305b9dd339a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:22.252943 1428 tablet_bootstrap.cc:654] T b7e50b9da893466486b668fcd0aac962 P 50e8c97891d74b38b5c130d3bcbabcb6: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:22.253160 1425 ts_tablet_manager.cc:1428] T 445539c2574245009333771c3464fcb9 P 75c4f59060974a3b823305b9dd339a7e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250627 13:52:22.253552 1425 tablet_bootstrap.cc:492] T b7e50b9da893466486b668fcd0aac962 P 75c4f59060974a3b823305b9dd339a7e: Bootstrap starting.
I20250627 13:52:22.255873 1425 tablet_bootstrap.cc:654] T b7e50b9da893466486b668fcd0aac962 P 75c4f59060974a3b823305b9dd339a7e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:22.257160 1430 tablet_bootstrap.cc:492] T b7e50b9da893466486b668fcd0aac962 P 5c28ff2fe9424379819d478fdd093cfc: No bootstrap required, opened a new log
I20250627 13:52:22.257418 1430 ts_tablet_manager.cc:1397] T b7e50b9da893466486b668fcd0aac962 P 5c28ff2fe9424379819d478fdd093cfc: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.013s
I20250627 13:52:22.258533 1430 raft_consensus.cc:357] T b7e50b9da893466486b668fcd0aac962 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:22.258828 1430 raft_consensus.cc:383] T b7e50b9da893466486b668fcd0aac962 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:22.258937 1430 raft_consensus.cc:738] T b7e50b9da893466486b668fcd0aac962 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c28ff2fe9424379819d478fdd093cfc, State: Initialized, Role: FOLLOWER
I20250627 13:52:22.259338 1430 consensus_queue.cc:260] T b7e50b9da893466486b668fcd0aac962 P 5c28ff2fe9424379819d478fdd093cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:22.260005 1430 ts_tablet_manager.cc:1428] T b7e50b9da893466486b668fcd0aac962 P 5c28ff2fe9424379819d478fdd093cfc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250627 13:52:22.263366 1426 raft_consensus.cc:357] T 46cbed417f5a4f2ebded566383ee3440 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } }
I20250627 13:52:22.263669 1426 raft_consensus.cc:383] T 46cbed417f5a4f2ebded566383ee3440 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:22.264218 1426 raft_consensus.cc:738] T 46cbed417f5a4f2ebded566383ee3440 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957a8a8be3f542f48e8824e581fd0f89, State: Initialized, Role: FOLLOWER
I20250627 13:52:22.264565 1426 consensus_queue.cc:260] T 46cbed417f5a4f2ebded566383ee3440 P 957a8a8be3f542f48e8824e581fd0f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } }
I20250627 13:52:22.265169 1426 ts_tablet_manager.cc:1428] T 46cbed417f5a4f2ebded566383ee3440 P 957a8a8be3f542f48e8824e581fd0f89: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250627 13:52:22.267477 1429 tablet_bootstrap.cc:492] T 46cbed417f5a4f2ebded566383ee3440 P 9f767210a91f44ae8b83b994131eee98: No bootstrap required, opened a new log
I20250627 13:52:22.267683 1429 ts_tablet_manager.cc:1397] T 46cbed417f5a4f2ebded566383ee3440 P 9f767210a91f44ae8b83b994131eee98: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.000s
I20250627 13:52:22.268150 1378 raft_consensus.cc:491] T 445539c2574245009333771c3464fcb9 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:22.268333 1378 raft_consensus.cc:513] T 445539c2574245009333771c3464fcb9 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:22.268503 1429 raft_consensus.cc:357] T 46cbed417f5a4f2ebded566383ee3440 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } }
I20250627 13:52:22.268689 1429 raft_consensus.cc:383] T 46cbed417f5a4f2ebded566383ee3440 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:22.268764 1429 raft_consensus.cc:738] T 46cbed417f5a4f2ebded566383ee3440 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f767210a91f44ae8b83b994131eee98, State: Initialized, Role: FOLLOWER
I20250627 13:52:22.268956 1429 consensus_queue.cc:260] T 46cbed417f5a4f2ebded566383ee3440 P 9f767210a91f44ae8b83b994131eee98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } }
I20250627 13:52:22.270046 1378 leader_election.cc:290] T 445539c2574245009333771c3464fcb9 P 75c4f59060974a3b823305b9dd339a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70:39887), 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68:34289)
I20250627 13:52:22.270164 1429 ts_tablet_manager.cc:1428] T 46cbed417f5a4f2ebded566383ee3440 P 9f767210a91f44ae8b83b994131eee98: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:52:22.271718 1427 tablet_bootstrap.cc:492] T 46cbed417f5a4f2ebded566383ee3440 P fa66786862da49ae826fb519c7d89b5e: No bootstrap required, opened a new log
I20250627 13:52:22.271867 1427 ts_tablet_manager.cc:1397] T 46cbed417f5a4f2ebded566383ee3440 P fa66786862da49ae826fb519c7d89b5e: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.000s
I20250627 13:52:22.272604 1427 raft_consensus.cc:357] T 46cbed417f5a4f2ebded566383ee3440 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } }
I20250627 13:52:22.272779 1427 raft_consensus.cc:383] T 46cbed417f5a4f2ebded566383ee3440 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:22.272850 1427 raft_consensus.cc:738] T 46cbed417f5a4f2ebded566383ee3440 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa66786862da49ae826fb519c7d89b5e, State: Initialized, Role: FOLLOWER
I20250627 13:52:22.273087 1427 consensus_queue.cc:260] T 46cbed417f5a4f2ebded566383ee3440 P fa66786862da49ae826fb519c7d89b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } }
I20250627 13:52:22.273587 1427 ts_tablet_manager.cc:1428] T 46cbed417f5a4f2ebded566383ee3440 P fa66786862da49ae826fb519c7d89b5e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:52:22.274811 1428 tablet_bootstrap.cc:492] T b7e50b9da893466486b668fcd0aac962 P 50e8c97891d74b38b5c130d3bcbabcb6: No bootstrap required, opened a new log
I20250627 13:52:22.275039 1428 ts_tablet_manager.cc:1397] T b7e50b9da893466486b668fcd0aac962 P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.004s
I20250627 13:52:22.276517 1428 raft_consensus.cc:357] T b7e50b9da893466486b668fcd0aac962 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:22.276959 1428 raft_consensus.cc:383] T b7e50b9da893466486b668fcd0aac962 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:22.277055 1428 raft_consensus.cc:738] T b7e50b9da893466486b668fcd0aac962 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e8c97891d74b38b5c130d3bcbabcb6, State: Initialized, Role: FOLLOWER
I20250627 13:52:22.277392 1428 consensus_queue.cc:260] T b7e50b9da893466486b668fcd0aac962 P 50e8c97891d74b38b5c130d3bcbabcb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:22.277904 1428 ts_tablet_manager.cc:1428] T b7e50b9da893466486b668fcd0aac962 P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250627 13:52:22.284099 1425 tablet_bootstrap.cc:492] T b7e50b9da893466486b668fcd0aac962 P 75c4f59060974a3b823305b9dd339a7e: No bootstrap required, opened a new log
I20250627 13:52:22.284742 1425 ts_tablet_manager.cc:1397] T b7e50b9da893466486b668fcd0aac962 P 75c4f59060974a3b823305b9dd339a7e: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.006s
I20250627 13:52:22.286458 1425 raft_consensus.cc:357] T b7e50b9da893466486b668fcd0aac962 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:22.286985 1425 raft_consensus.cc:383] T b7e50b9da893466486b668fcd0aac962 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:22.287185 1425 raft_consensus.cc:738] T b7e50b9da893466486b668fcd0aac962 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c4f59060974a3b823305b9dd339a7e, State: Initialized, Role: FOLLOWER
I20250627 13:52:22.287511 1425 consensus_queue.cc:260] T b7e50b9da893466486b668fcd0aac962 P 75c4f59060974a3b823305b9dd339a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:22.288347 1425 ts_tablet_manager.cc:1428] T b7e50b9da893466486b668fcd0aac962 P 75c4f59060974a3b823305b9dd339a7e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250627 13:52:22.295313 1254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "445539c2574245009333771c3464fcb9" candidate_uuid: "75c4f59060974a3b823305b9dd339a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c28ff2fe9424379819d478fdd093cfc" is_pre_election: true
I20250627 13:52:22.295771 1254 raft_consensus.cc:2466] T 445539c2574245009333771c3464fcb9 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75c4f59060974a3b823305b9dd339a7e in term 0.
I20250627 13:52:22.296449 664 leader_election.cc:304] T 445539c2574245009333771c3464fcb9 P 75c4f59060974a3b823305b9dd339a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c28ff2fe9424379819d478fdd093cfc, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:22.297416 1378 raft_consensus.cc:2802] T 445539c2574245009333771c3464fcb9 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:22.297550 1378 raft_consensus.cc:491] T 445539c2574245009333771c3464fcb9 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:22.297677 1378 raft_consensus.cc:3058] T 445539c2574245009333771c3464fcb9 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:22.300088 1378 raft_consensus.cc:513] T 445539c2574245009333771c3464fcb9 P 75c4f59060974a3b823305b9dd339a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:22.300796 1378 leader_election.cc:290] T 445539c2574245009333771c3464fcb9 P 75c4f59060974a3b823305b9dd339a7e [CANDIDATE]: Term 1 election: Requested vote from peers 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70:39887), 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68:34289)
I20250627 13:52:22.301249 1254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "445539c2574245009333771c3464fcb9" candidate_uuid: "75c4f59060974a3b823305b9dd339a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c28ff2fe9424379819d478fdd093cfc"
I20250627 13:52:22.301467 1254 raft_consensus.cc:3058] T 445539c2574245009333771c3464fcb9 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:22.301630 991 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "445539c2574245009333771c3464fcb9" candidate_uuid: "75c4f59060974a3b823305b9dd339a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" is_pre_election: true
I20250627 13:52:22.301764 992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "445539c2574245009333771c3464fcb9" candidate_uuid: "75c4f59060974a3b823305b9dd339a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e8c97891d74b38b5c130d3bcbabcb6"
I20250627 13:52:22.301903 991 raft_consensus.cc:2466] T 445539c2574245009333771c3464fcb9 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75c4f59060974a3b823305b9dd339a7e in term 0.
I20250627 13:52:22.302296 1349 raft_consensus.cc:491] T b7e50b9da893466486b668fcd0aac962 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:22.302453 1349 raft_consensus.cc:513] T b7e50b9da893466486b668fcd0aac962 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:22.303156 1349 leader_election.cc:290] T b7e50b9da893466486b668fcd0aac962 P 5c28ff2fe9424379819d478fdd093cfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499), 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68:34289)
I20250627 13:52:22.303766 730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b7e50b9da893466486b668fcd0aac962" candidate_uuid: "5c28ff2fe9424379819d478fdd093cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c4f59060974a3b823305b9dd339a7e" is_pre_election: true
I20250627 13:52:22.303983 730 raft_consensus.cc:2466] T b7e50b9da893466486b668fcd0aac962 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c28ff2fe9424379819d478fdd093cfc in term 0.
I20250627 13:52:22.304186 1254 raft_consensus.cc:2466] T 445539c2574245009333771c3464fcb9 P 5c28ff2fe9424379819d478fdd093cfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75c4f59060974a3b823305b9dd339a7e in term 1.
I20250627 13:52:22.304436 1190 leader_election.cc:304] T b7e50b9da893466486b668fcd0aac962 P 5c28ff2fe9424379819d478fdd093cfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c28ff2fe9424379819d478fdd093cfc, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:22.304764 1349 raft_consensus.cc:2802] T b7e50b9da893466486b668fcd0aac962 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:22.304845 664 leader_election.cc:304] T 445539c2574245009333771c3464fcb9 P 75c4f59060974a3b823305b9dd339a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c28ff2fe9424379819d478fdd093cfc, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:22.304909 1349 raft_consensus.cc:491] T b7e50b9da893466486b668fcd0aac962 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:22.305037 1349 raft_consensus.cc:3058] T b7e50b9da893466486b668fcd0aac962 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:22.305114 1378 raft_consensus.cc:2802] T 445539c2574245009333771c3464fcb9 P 75c4f59060974a3b823305b9dd339a7e [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:22.305150 1352 raft_consensus.cc:491] T 46cbed417f5a4f2ebded566383ee3440 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:22.305320 1352 raft_consensus.cc:513] T 46cbed417f5a4f2ebded566383ee3440 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } }
I20250627 13:52:22.306214 1352 leader_election.cc:290] T 46cbed417f5a4f2ebded566383ee3440 P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023), fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747)
I20250627 13:52:22.307153 597 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46cbed417f5a4f2ebded566383ee3440" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98" is_pre_election: true
I20250627 13:52:22.307452 597 raft_consensus.cc:2466] T 46cbed417f5a4f2ebded566383ee3440 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 0.
I20250627 13:52:22.308110 1056 leader_election.cc:304] T 46cbed417f5a4f2ebded566383ee3440 P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, 9f767210a91f44ae8b83b994131eee98; no voters:
I20250627 13:52:22.309326 1349 raft_consensus.cc:513] T b7e50b9da893466486b668fcd0aac962 P 5c28ff2fe9424379819d478fdd093cfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:22.309605 861 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46cbed417f5a4f2ebded566383ee3440" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e" is_pre_election: true
I20250627 13:52:22.309873 861 raft_consensus.cc:2466] T 46cbed417f5a4f2ebded566383ee3440 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 0.
I20250627 13:52:22.310178 1349 leader_election.cc:290] T b7e50b9da893466486b668fcd0aac962 P 5c28ff2fe9424379819d478fdd093cfc [CANDIDATE]: Term 1 election: Requested vote from peers 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499), 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68:34289)
I20250627 13:52:22.310441 730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b7e50b9da893466486b668fcd0aac962" candidate_uuid: "5c28ff2fe9424379819d478fdd093cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c4f59060974a3b823305b9dd339a7e"
I20250627 13:52:22.310525 1352 raft_consensus.cc:2802] T 46cbed417f5a4f2ebded566383ee3440 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:22.310609 1352 raft_consensus.cc:491] T 46cbed417f5a4f2ebded566383ee3440 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:22.310642 730 raft_consensus.cc:3058] T b7e50b9da893466486b668fcd0aac962 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:22.310719 1352 raft_consensus.cc:3058] T 46cbed417f5a4f2ebded566383ee3440 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:22.313516 730 raft_consensus.cc:2466] T b7e50b9da893466486b668fcd0aac962 P 75c4f59060974a3b823305b9dd339a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c28ff2fe9424379819d478fdd093cfc in term 1.
I20250627 13:52:22.313793 1352 raft_consensus.cc:513] T 46cbed417f5a4f2ebded566383ee3440 P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } }
I20250627 13:52:22.313987 1190 leader_election.cc:304] T b7e50b9da893466486b668fcd0aac962 P 5c28ff2fe9424379819d478fdd093cfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c28ff2fe9424379819d478fdd093cfc, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:22.314251 1349 raft_consensus.cc:2802] T b7e50b9da893466486b668fcd0aac962 P 5c28ff2fe9424379819d478fdd093cfc [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:22.314522 1349 raft_consensus.cc:695] T b7e50b9da893466486b668fcd0aac962 P 5c28ff2fe9424379819d478fdd093cfc [term 1 LEADER]: Becoming Leader. State: Replica: 5c28ff2fe9424379819d478fdd093cfc, State: Running, Role: LEADER
I20250627 13:52:22.314837 1349 consensus_queue.cc:237] T b7e50b9da893466486b668fcd0aac962 P 5c28ff2fe9424379819d478fdd093cfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:22.314980 597 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46cbed417f5a4f2ebded566383ee3440" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98"
I20250627 13:52:22.315178 597 raft_consensus.cc:3058] T 46cbed417f5a4f2ebded566383ee3440 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:22.318027 1352 leader_election.cc:290] T 46cbed417f5a4f2ebded566383ee3440 P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 election: Requested vote from peers 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023), fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747)
I20250627 13:52:22.318598 597 raft_consensus.cc:2466] T 46cbed417f5a4f2ebded566383ee3440 P 9f767210a91f44ae8b83b994131eee98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 1.
I20250627 13:52:22.319132 1056 leader_election.cc:304] T 46cbed417f5a4f2ebded566383ee3440 P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, 9f767210a91f44ae8b83b994131eee98; no voters:
I20250627 13:52:22.319496 1352 raft_consensus.cc:2802] T 46cbed417f5a4f2ebded566383ee3440 P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:22.319674 1352 raft_consensus.cc:695] T 46cbed417f5a4f2ebded566383ee3440 P 957a8a8be3f542f48e8824e581fd0f89 [term 1 LEADER]: Becoming Leader. State: Replica: 957a8a8be3f542f48e8824e581fd0f89, State: Running, Role: LEADER
I20250627 13:52:22.319779 1378 raft_consensus.cc:695] T 445539c2574245009333771c3464fcb9 P 75c4f59060974a3b823305b9dd339a7e [term 1 LEADER]: Becoming Leader. State: Replica: 75c4f59060974a3b823305b9dd339a7e, State: Running, Role: LEADER
I20250627 13:52:22.319918 1352 consensus_queue.cc:237] T 46cbed417f5a4f2ebded566383ee3440 P 957a8a8be3f542f48e8824e581fd0f89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } }
I20250627 13:52:22.320269 1378 consensus_queue.cc:237] T 445539c2574245009333771c3464fcb9 P 75c4f59060974a3b823305b9dd339a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:22.320822 454 catalog_manager.cc:5582] T b7e50b9da893466486b668fcd0aac962 P 5c28ff2fe9424379819d478fdd093cfc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70). New cstate: current_term: 1 leader_uuid: "5c28ff2fe9424379819d478fdd093cfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } health_report { overall_health: UNKNOWN } } }
I20250627 13:52:22.321403 861 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46cbed417f5a4f2ebded566383ee3440" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e"
I20250627 13:52:22.321601 861 raft_consensus.cc:3058] T 46cbed417f5a4f2ebded566383ee3440 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:22.325073 861 raft_consensus.cc:2466] T 46cbed417f5a4f2ebded566383ee3440 P fa66786862da49ae826fb519c7d89b5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 1.
I20250627 13:52:22.327044 992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b7e50b9da893466486b668fcd0aac962" candidate_uuid: "5c28ff2fe9424379819d478fdd093cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" is_pre_election: true
I20250627 13:52:22.327128 991 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b7e50b9da893466486b668fcd0aac962" candidate_uuid: "5c28ff2fe9424379819d478fdd093cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e8c97891d74b38b5c130d3bcbabcb6"
I20250627 13:52:22.327286 991 raft_consensus.cc:3058] T b7e50b9da893466486b668fcd0aac962 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:22.329392 991 raft_consensus.cc:2466] T b7e50b9da893466486b668fcd0aac962 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c28ff2fe9424379819d478fdd093cfc in term 1.
I20250627 13:52:22.332212 454 catalog_manager.cc:5582] T 445539c2574245009333771c3464fcb9 P 75c4f59060974a3b823305b9dd339a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to 75c4f59060974a3b823305b9dd339a7e (127.0.103.66). New cstate: current_term: 1 leader_uuid: "75c4f59060974a3b823305b9dd339a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } health_report { overall_health: HEALTHY } } }
I20250627 13:52:22.339754 454 catalog_manager.cc:5582] T 46cbed417f5a4f2ebded566383ee3440 P 957a8a8be3f542f48e8824e581fd0f89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69). New cstate: current_term: 1 leader_uuid: "957a8a8be3f542f48e8824e581fd0f89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } health_report { overall_health: UNKNOWN } } }
W20250627 13:52:22.686206 1448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:22.686560 1448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:22.694092 1448 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20250627 13:52:22.731593 1352 consensus_queue.cc:1035] T 46cbed417f5a4f2ebded566383ee3440 P 957a8a8be3f542f48e8824e581fd0f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:22.740211 1349 consensus_queue.cc:1035] T b7e50b9da893466486b668fcd0aac962 P 5c28ff2fe9424379819d478fdd093cfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:22.746459 1394 consensus_queue.cc:1035] T 46cbed417f5a4f2ebded566383ee3440 P 957a8a8be3f542f48e8824e581fd0f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:22.762660 1342 consensus_queue.cc:1035] T b7e50b9da893466486b668fcd0aac962 P 5c28ff2fe9424379819d478fdd093cfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:22.793080 991 raft_consensus.cc:3058] T 445539c2574245009333771c3464fcb9 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:22.801541 1378 consensus_queue.cc:1035] T 445539c2574245009333771c3464fcb9 P 75c4f59060974a3b823305b9dd339a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:22.821604 1444 consensus_queue.cc:1035] T 445539c2574245009333771c3464fcb9 P 75c4f59060974a3b823305b9dd339a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 267.956 ms
time per row: 0.133978 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20250627 13:52:24.109156 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49852:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250627 13:52:24.118276 1103 tablet_service.cc:1468] Processing CreateTablet for tablet f3a32a4bc6ad45ebacea4f92ebabeb95 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e4125d5eed0141d7b20100d295842649]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:24.118754 1103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3a32a4bc6ad45ebacea4f92ebabeb95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:24.119009 972 tablet_service.cc:1468] Processing CreateTablet for tablet fc3d1ee303e64241930642d9f13a3fd0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e4125d5eed0141d7b20100d295842649]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:24.119277 710 tablet_service.cc:1468] Processing CreateTablet for tablet fc3d1ee303e64241930642d9f13a3fd0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e4125d5eed0141d7b20100d295842649]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:24.119415 972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc3d1ee303e64241930642d9f13a3fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:24.119657 710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc3d1ee303e64241930642d9f13a3fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:24.120714 1234 tablet_service.cc:1468] Processing CreateTablet for tablet fc3d1ee303e64241930642d9f13a3fd0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e4125d5eed0141d7b20100d295842649]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:24.121073 1234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc3d1ee303e64241930642d9f13a3fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:24.122135 840 tablet_service.cc:1468] Processing CreateTablet for tablet f3a32a4bc6ad45ebacea4f92ebabeb95 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e4125d5eed0141d7b20100d295842649]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:24.122526 840 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3a32a4bc6ad45ebacea4f92ebabeb95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:24.126055 1102 tablet_service.cc:1468] Processing CreateTablet for tablet 5626734eb5a8437b97dd89f03c131ffa (DEFAULT_TABLE table=rebalance_test_table_2 [id=e4125d5eed0141d7b20100d295842649]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:24.126451 1102 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5626734eb5a8437b97dd89f03c131ffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:24.122138 841 tablet_service.cc:1468] Processing CreateTablet for tablet 5626734eb5a8437b97dd89f03c131ffa (DEFAULT_TABLE table=rebalance_test_table_2 [id=e4125d5eed0141d7b20100d295842649]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:24.130460 578 tablet_service.cc:1468] Processing CreateTablet for tablet f3a32a4bc6ad45ebacea4f92ebabeb95 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e4125d5eed0141d7b20100d295842649]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:24.130750 841 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5626734eb5a8437b97dd89f03c131ffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:24.130985 578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3a32a4bc6ad45ebacea4f92ebabeb95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:24.134204 577 tablet_service.cc:1468] Processing CreateTablet for tablet 5626734eb5a8437b97dd89f03c131ffa (DEFAULT_TABLE table=rebalance_test_table_2 [id=e4125d5eed0141d7b20100d295842649]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:24.134675 577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5626734eb5a8437b97dd89f03c131ffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:24.136839 1516 tablet_bootstrap.cc:492] T fc3d1ee303e64241930642d9f13a3fd0 P 5c28ff2fe9424379819d478fdd093cfc: Bootstrap starting.
I20250627 13:52:24.139448 1516 tablet_bootstrap.cc:654] T fc3d1ee303e64241930642d9f13a3fd0 P 5c28ff2fe9424379819d478fdd093cfc: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:24.148164 1518 tablet_bootstrap.cc:492] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 9f767210a91f44ae8b83b994131eee98: Bootstrap starting.
I20250627 13:52:24.149760 1516 tablet_bootstrap.cc:492] T fc3d1ee303e64241930642d9f13a3fd0 P 5c28ff2fe9424379819d478fdd093cfc: No bootstrap required, opened a new log
I20250627 13:52:24.149614 1520 tablet_bootstrap.cc:492] T 5626734eb5a8437b97dd89f03c131ffa P 957a8a8be3f542f48e8824e581fd0f89: Bootstrap starting.
I20250627 13:52:24.149924 1516 ts_tablet_manager.cc:1397] T fc3d1ee303e64241930642d9f13a3fd0 P 5c28ff2fe9424379819d478fdd093cfc: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20250627 13:52:24.150660 1518 tablet_bootstrap.cc:654] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 9f767210a91f44ae8b83b994131eee98: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:24.150835 1516 raft_consensus.cc:357] T fc3d1ee303e64241930642d9f13a3fd0 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:24.151024 1516 raft_consensus.cc:383] T fc3d1ee303e64241930642d9f13a3fd0 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:24.151125 1516 raft_consensus.cc:738] T fc3d1ee303e64241930642d9f13a3fd0 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c28ff2fe9424379819d478fdd093cfc, State: Initialized, Role: FOLLOWER
I20250627 13:52:24.151355 1516 consensus_queue.cc:260] T fc3d1ee303e64241930642d9f13a3fd0 P 5c28ff2fe9424379819d478fdd093cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:24.151479 1520 tablet_bootstrap.cc:654] T 5626734eb5a8437b97dd89f03c131ffa P 957a8a8be3f542f48e8824e581fd0f89: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:24.151981 1516 ts_tablet_manager.cc:1428] T fc3d1ee303e64241930642d9f13a3fd0 P 5c28ff2fe9424379819d478fdd093cfc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250627 13:52:24.152407 1514 tablet_bootstrap.cc:492] T fc3d1ee303e64241930642d9f13a3fd0 P 50e8c97891d74b38b5c130d3bcbabcb6: Bootstrap starting.
I20250627 13:52:24.154194 1514 tablet_bootstrap.cc:654] T fc3d1ee303e64241930642d9f13a3fd0 P 50e8c97891d74b38b5c130d3bcbabcb6: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:24.159969 1514 tablet_bootstrap.cc:492] T fc3d1ee303e64241930642d9f13a3fd0 P 50e8c97891d74b38b5c130d3bcbabcb6: No bootstrap required, opened a new log
I20250627 13:52:24.160815 1514 ts_tablet_manager.cc:1397] T fc3d1ee303e64241930642d9f13a3fd0 P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20250627 13:52:24.161795 1520 tablet_bootstrap.cc:492] T 5626734eb5a8437b97dd89f03c131ffa P 957a8a8be3f542f48e8824e581fd0f89: No bootstrap required, opened a new log
I20250627 13:52:24.161827 1514 raft_consensus.cc:357] T fc3d1ee303e64241930642d9f13a3fd0 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:24.162019 1520 ts_tablet_manager.cc:1397] T 5626734eb5a8437b97dd89f03c131ffa P 957a8a8be3f542f48e8824e581fd0f89: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.009s
I20250627 13:52:24.162909 1520 raft_consensus.cc:357] T 5626734eb5a8437b97dd89f03c131ffa P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } }
I20250627 13:52:24.163136 1520 raft_consensus.cc:383] T 5626734eb5a8437b97dd89f03c131ffa P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:24.163296 1520 raft_consensus.cc:738] T 5626734eb5a8437b97dd89f03c131ffa P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957a8a8be3f542f48e8824e581fd0f89, State: Initialized, Role: FOLLOWER
I20250627 13:52:24.163606 1520 consensus_queue.cc:260] T 5626734eb5a8437b97dd89f03c131ffa P 957a8a8be3f542f48e8824e581fd0f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } }
I20250627 13:52:24.163774 1518 tablet_bootstrap.cc:492] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 9f767210a91f44ae8b83b994131eee98: No bootstrap required, opened a new log
I20250627 13:52:24.163952 1518 ts_tablet_manager.cc:1397] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 9f767210a91f44ae8b83b994131eee98: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.000s
I20250627 13:52:24.164804 1518 raft_consensus.cc:357] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:24.165046 1518 raft_consensus.cc:383] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:24.165186 1518 raft_consensus.cc:738] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f767210a91f44ae8b83b994131eee98, State: Initialized, Role: FOLLOWER
I20250627 13:52:24.165483 1518 consensus_queue.cc:260] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 9f767210a91f44ae8b83b994131eee98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:24.165936 1514 raft_consensus.cc:383] T fc3d1ee303e64241930642d9f13a3fd0 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:24.166081 1514 raft_consensus.cc:738] T fc3d1ee303e64241930642d9f13a3fd0 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e8c97891d74b38b5c130d3bcbabcb6, State: Initialized, Role: FOLLOWER
I20250627 13:52:24.166384 1514 consensus_queue.cc:260] T fc3d1ee303e64241930642d9f13a3fd0 P 50e8c97891d74b38b5c130d3bcbabcb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:24.166886 1520 ts_tablet_manager.cc:1428] T 5626734eb5a8437b97dd89f03c131ffa P 957a8a8be3f542f48e8824e581fd0f89: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250627 13:52:24.153781 1519 tablet_bootstrap.cc:492] T 5626734eb5a8437b97dd89f03c131ffa P fa66786862da49ae826fb519c7d89b5e: Bootstrap starting.
I20250627 13:52:24.170888 1519 tablet_bootstrap.cc:654] T 5626734eb5a8437b97dd89f03c131ffa P fa66786862da49ae826fb519c7d89b5e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:24.171617 1520 tablet_bootstrap.cc:492] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 957a8a8be3f542f48e8824e581fd0f89: Bootstrap starting.
I20250627 13:52:24.173841 1520 tablet_bootstrap.cc:654] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 957a8a8be3f542f48e8824e581fd0f89: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:24.175261 1514 ts_tablet_manager.cc:1428] T fc3d1ee303e64241930642d9f13a3fd0 P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20250627 13:52:24.176776 1518 ts_tablet_manager.cc:1428] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 9f767210a91f44ae8b83b994131eee98: Time spent starting tablet: real 0.013s user 0.001s sys 0.004s
I20250627 13:52:24.177121 1518 tablet_bootstrap.cc:492] T 5626734eb5a8437b97dd89f03c131ffa P 9f767210a91f44ae8b83b994131eee98: Bootstrap starting.
I20250627 13:52:24.178432 1515 tablet_bootstrap.cc:492] T fc3d1ee303e64241930642d9f13a3fd0 P 75c4f59060974a3b823305b9dd339a7e: Bootstrap starting.
I20250627 13:52:24.179180 1518 tablet_bootstrap.cc:654] T 5626734eb5a8437b97dd89f03c131ffa P 9f767210a91f44ae8b83b994131eee98: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:24.180797 1515 tablet_bootstrap.cc:654] T fc3d1ee303e64241930642d9f13a3fd0 P 75c4f59060974a3b823305b9dd339a7e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:24.181502 1519 tablet_bootstrap.cc:492] T 5626734eb5a8437b97dd89f03c131ffa P fa66786862da49ae826fb519c7d89b5e: No bootstrap required, opened a new log
I20250627 13:52:24.181715 1519 ts_tablet_manager.cc:1397] T 5626734eb5a8437b97dd89f03c131ffa P fa66786862da49ae826fb519c7d89b5e: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.000s
I20250627 13:52:24.182638 1519 raft_consensus.cc:357] T 5626734eb5a8437b97dd89f03c131ffa P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } }
I20250627 13:52:24.182874 1519 raft_consensus.cc:383] T 5626734eb5a8437b97dd89f03c131ffa P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:24.182994 1519 raft_consensus.cc:738] T 5626734eb5a8437b97dd89f03c131ffa P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa66786862da49ae826fb519c7d89b5e, State: Initialized, Role: FOLLOWER
I20250627 13:52:24.183274 1519 consensus_queue.cc:260] T 5626734eb5a8437b97dd89f03c131ffa P fa66786862da49ae826fb519c7d89b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } }
I20250627 13:52:24.191864 1515 tablet_bootstrap.cc:492] T fc3d1ee303e64241930642d9f13a3fd0 P 75c4f59060974a3b823305b9dd339a7e: No bootstrap required, opened a new log
I20250627 13:52:24.191882 1519 ts_tablet_manager.cc:1428] T 5626734eb5a8437b97dd89f03c131ffa P fa66786862da49ae826fb519c7d89b5e: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20250627 13:52:24.192020 1515 ts_tablet_manager.cc:1397] T fc3d1ee303e64241930642d9f13a3fd0 P 75c4f59060974a3b823305b9dd339a7e: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.003s
I20250627 13:52:24.192739 1515 raft_consensus.cc:357] T fc3d1ee303e64241930642d9f13a3fd0 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:24.192906 1515 raft_consensus.cc:383] T fc3d1ee303e64241930642d9f13a3fd0 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:24.192982 1515 raft_consensus.cc:738] T fc3d1ee303e64241930642d9f13a3fd0 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c4f59060974a3b823305b9dd339a7e, State: Initialized, Role: FOLLOWER
I20250627 13:52:24.193238 1519 tablet_bootstrap.cc:492] T f3a32a4bc6ad45ebacea4f92ebabeb95 P fa66786862da49ae826fb519c7d89b5e: Bootstrap starting.
I20250627 13:52:24.193177 1515 consensus_queue.cc:260] T fc3d1ee303e64241930642d9f13a3fd0 P 75c4f59060974a3b823305b9dd339a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:24.193672 1515 ts_tablet_manager.cc:1428] T fc3d1ee303e64241930642d9f13a3fd0 P 75c4f59060974a3b823305b9dd339a7e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250627 13:52:24.197628 1520 tablet_bootstrap.cc:492] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 957a8a8be3f542f48e8824e581fd0f89: No bootstrap required, opened a new log
I20250627 13:52:24.197839 1520 ts_tablet_manager.cc:1397] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 957a8a8be3f542f48e8824e581fd0f89: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.006s
I20250627 13:52:24.198518 1518 tablet_bootstrap.cc:492] T 5626734eb5a8437b97dd89f03c131ffa P 9f767210a91f44ae8b83b994131eee98: No bootstrap required, opened a new log
I20250627 13:52:24.198758 1518 ts_tablet_manager.cc:1397] T 5626734eb5a8437b97dd89f03c131ffa P 9f767210a91f44ae8b83b994131eee98: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.005s
I20250627 13:52:24.199138 1519 tablet_bootstrap.cc:654] T f3a32a4bc6ad45ebacea4f92ebabeb95 P fa66786862da49ae826fb519c7d89b5e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:24.199502 1518 raft_consensus.cc:357] T 5626734eb5a8437b97dd89f03c131ffa P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } }
I20250627 13:52:24.199748 1518 raft_consensus.cc:383] T 5626734eb5a8437b97dd89f03c131ffa P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:24.199846 1518 raft_consensus.cc:738] T 5626734eb5a8437b97dd89f03c131ffa P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f767210a91f44ae8b83b994131eee98, State: Initialized, Role: FOLLOWER
I20250627 13:52:24.200099 1520 raft_consensus.cc:357] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:24.200182 1518 consensus_queue.cc:260] T 5626734eb5a8437b97dd89f03c131ffa P 9f767210a91f44ae8b83b994131eee98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } }
I20250627 13:52:24.200376 1520 raft_consensus.cc:383] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:24.200520 1520 raft_consensus.cc:738] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957a8a8be3f542f48e8824e581fd0f89, State: Initialized, Role: FOLLOWER
I20250627 13:52:24.200841 1518 ts_tablet_manager.cc:1428] T 5626734eb5a8437b97dd89f03c131ffa P 9f767210a91f44ae8b83b994131eee98: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250627 13:52:24.200888 1520 consensus_queue.cc:260] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 957a8a8be3f542f48e8824e581fd0f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:24.203569 1519 tablet_bootstrap.cc:492] T f3a32a4bc6ad45ebacea4f92ebabeb95 P fa66786862da49ae826fb519c7d89b5e: No bootstrap required, opened a new log
I20250627 13:52:24.203810 1519 ts_tablet_manager.cc:1397] T f3a32a4bc6ad45ebacea4f92ebabeb95 P fa66786862da49ae826fb519c7d89b5e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20250627 13:52:24.204643 1519 raft_consensus.cc:357] T f3a32a4bc6ad45ebacea4f92ebabeb95 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:24.204871 1519 raft_consensus.cc:383] T f3a32a4bc6ad45ebacea4f92ebabeb95 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:24.205013 1519 raft_consensus.cc:738] T f3a32a4bc6ad45ebacea4f92ebabeb95 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa66786862da49ae826fb519c7d89b5e, State: Initialized, Role: FOLLOWER
I20250627 13:52:24.205257 1519 consensus_queue.cc:260] T f3a32a4bc6ad45ebacea4f92ebabeb95 P fa66786862da49ae826fb519c7d89b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:24.205449 1520 ts_tablet_manager.cc:1428] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 957a8a8be3f542f48e8824e581fd0f89: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250627 13:52:24.206262 1519 ts_tablet_manager.cc:1428] T f3a32a4bc6ad45ebacea4f92ebabeb95 P fa66786862da49ae826fb519c7d89b5e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:52:24.325706 1530 raft_consensus.cc:491] T 5626734eb5a8437b97dd89f03c131ffa P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:24.325891 1530 raft_consensus.cc:513] T 5626734eb5a8437b97dd89f03c131ffa P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } }
I20250627 13:52:24.327266 1530 leader_election.cc:290] T 5626734eb5a8437b97dd89f03c131ffa P fa66786862da49ae826fb519c7d89b5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69:42325), 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023)
I20250627 13:52:24.330633 1524 raft_consensus.cc:491] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:24.330875 1524 raft_consensus.cc:513] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:24.333026 1524 leader_election.cc:290] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 9f767210a91f44ae8b83b994131eee98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69:42325), fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747)
I20250627 13:52:24.339735 1352 raft_consensus.cc:491] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:24.340020 1352 raft_consensus.cc:513] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:24.340888 1352 leader_election.cc:290] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747), 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023)
I20250627 13:52:24.341739 860 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3a32a4bc6ad45ebacea4f92ebabeb95" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e" is_pre_election: true
I20250627 13:52:24.342007 860 raft_consensus.cc:2466] T f3a32a4bc6ad45ebacea4f92ebabeb95 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 0.
I20250627 13:52:24.342643 1058 leader_election.cc:304] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, fa66786862da49ae826fb519c7d89b5e; no voters:
I20250627 13:52:24.343072 1352 raft_consensus.cc:2802] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:24.343178 1352 raft_consensus.cc:491] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:24.343288 1352 raft_consensus.cc:3058] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:24.346655 1352 raft_consensus.cc:513] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:24.347693 860 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3a32a4bc6ad45ebacea4f92ebabeb95" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e"
I20250627 13:52:24.347896 860 raft_consensus.cc:3058] T f3a32a4bc6ad45ebacea4f92ebabeb95 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:24.347934 1352 leader_election.cc:290] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 election: Requested vote from peers fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747), 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023)
I20250627 13:52:24.351312 860 raft_consensus.cc:2466] T f3a32a4bc6ad45ebacea4f92ebabeb95 P fa66786862da49ae826fb519c7d89b5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 1.
I20250627 13:52:24.351855 1058 leader_election.cc:304] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, fa66786862da49ae826fb519c7d89b5e; no voters:
I20250627 13:52:24.352229 1352 raft_consensus.cc:2802] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:24.352712 1352 raft_consensus.cc:695] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 957a8a8be3f542f48e8824e581fd0f89 [term 1 LEADER]: Becoming Leader. State: Replica: 957a8a8be3f542f48e8824e581fd0f89, State: Running, Role: LEADER
I20250627 13:52:24.353075 1352 consensus_queue.cc:237] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 957a8a8be3f542f48e8824e581fd0f89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:24.358189 447 catalog_manager.cc:5582] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 957a8a8be3f542f48e8824e581fd0f89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69). New cstate: current_term: 1 leader_uuid: "957a8a8be3f542f48e8824e581fd0f89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } health_report { overall_health: UNKNOWN } } }
I20250627 13:52:24.359606 1123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5626734eb5a8437b97dd89f03c131ffa" candidate_uuid: "fa66786862da49ae826fb519c7d89b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957a8a8be3f542f48e8824e581fd0f89" is_pre_election: true
I20250627 13:52:24.359927 1123 raft_consensus.cc:2466] T 5626734eb5a8437b97dd89f03c131ffa P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa66786862da49ae826fb519c7d89b5e in term 0.
I20250627 13:52:24.360474 597 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3a32a4bc6ad45ebacea4f92ebabeb95" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98" is_pre_election: true
I20250627 13:52:24.360738 597 raft_consensus.cc:2466] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 0.
I20250627 13:52:24.361322 597 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3a32a4bc6ad45ebacea4f92ebabeb95" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98"
I20250627 13:52:24.361514 597 raft_consensus.cc:3058] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:24.362360 797 leader_election.cc:304] T 5626734eb5a8437b97dd89f03c131ffa P fa66786862da49ae826fb519c7d89b5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, fa66786862da49ae826fb519c7d89b5e; no voters:
I20250627 13:52:24.362728 1530 raft_consensus.cc:2802] T 5626734eb5a8437b97dd89f03c131ffa P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:24.362835 1530 raft_consensus.cc:491] T 5626734eb5a8437b97dd89f03c131ffa P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:24.362907 1530 raft_consensus.cc:3058] T 5626734eb5a8437b97dd89f03c131ffa P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:24.365465 597 raft_consensus.cc:2466] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 9f767210a91f44ae8b83b994131eee98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 1.
I20250627 13:52:24.365695 1530 raft_consensus.cc:513] T 5626734eb5a8437b97dd89f03c131ffa P fa66786862da49ae826fb519c7d89b5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } }
I20250627 13:52:24.367609 1123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5626734eb5a8437b97dd89f03c131ffa" candidate_uuid: "fa66786862da49ae826fb519c7d89b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957a8a8be3f542f48e8824e581fd0f89"
I20250627 13:52:24.367786 1123 raft_consensus.cc:3058] T 5626734eb5a8437b97dd89f03c131ffa P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:24.370841 1123 raft_consensus.cc:2466] T 5626734eb5a8437b97dd89f03c131ffa P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa66786862da49ae826fb519c7d89b5e in term 1.
I20250627 13:52:24.371237 797 leader_election.cc:304] T 5626734eb5a8437b97dd89f03c131ffa P fa66786862da49ae826fb519c7d89b5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, fa66786862da49ae826fb519c7d89b5e; no voters:
I20250627 13:52:24.371840 1530 leader_election.cc:290] T 5626734eb5a8437b97dd89f03c131ffa P fa66786862da49ae826fb519c7d89b5e [CANDIDATE]: Term 1 election: Requested vote from peers 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69:42325), 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023)
I20250627 13:52:24.371937 1122 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3a32a4bc6ad45ebacea4f92ebabeb95" candidate_uuid: "9f767210a91f44ae8b83b994131eee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957a8a8be3f542f48e8824e581fd0f89" is_pre_election: true
I20250627 13:52:24.372202 1530 raft_consensus.cc:2802] T 5626734eb5a8437b97dd89f03c131ffa P fa66786862da49ae826fb519c7d89b5e [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:24.372496 1530 raft_consensus.cc:695] T 5626734eb5a8437b97dd89f03c131ffa P fa66786862da49ae826fb519c7d89b5e [term 1 LEADER]: Becoming Leader. State: Replica: fa66786862da49ae826fb519c7d89b5e, State: Running, Role: LEADER
I20250627 13:52:24.373229 1530 consensus_queue.cc:237] T 5626734eb5a8437b97dd89f03c131ffa P fa66786862da49ae826fb519c7d89b5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } }
I20250627 13:52:24.378178 597 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5626734eb5a8437b97dd89f03c131ffa" candidate_uuid: "fa66786862da49ae826fb519c7d89b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98" is_pre_election: true
I20250627 13:52:24.378252 598 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5626734eb5a8437b97dd89f03c131ffa" candidate_uuid: "fa66786862da49ae826fb519c7d89b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98"
I20250627 13:52:24.378440 597 raft_consensus.cc:2466] T 5626734eb5a8437b97dd89f03c131ffa P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa66786862da49ae826fb519c7d89b5e in term 0.
I20250627 13:52:24.378228 447 catalog_manager.cc:5582] T 5626734eb5a8437b97dd89f03c131ffa P fa66786862da49ae826fb519c7d89b5e reported cstate change: term changed from 0 to 1, leader changed from <none> to fa66786862da49ae826fb519c7d89b5e (127.0.103.67). New cstate: current_term: 1 leader_uuid: "fa66786862da49ae826fb519c7d89b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } health_report { overall_health: HEALTHY } } }
I20250627 13:52:24.381204 860 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3a32a4bc6ad45ebacea4f92ebabeb95" candidate_uuid: "9f767210a91f44ae8b83b994131eee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e" is_pre_election: true
I20250627 13:52:24.381465 860 raft_consensus.cc:2391] T f3a32a4bc6ad45ebacea4f92ebabeb95 P fa66786862da49ae826fb519c7d89b5e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f767210a91f44ae8b83b994131eee98 in current term 1: Already voted for candidate 957a8a8be3f542f48e8824e581fd0f89 in this term.
I20250627 13:52:24.381896 533 leader_election.cc:304] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 9f767210a91f44ae8b83b994131eee98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f767210a91f44ae8b83b994131eee98; no voters: 957a8a8be3f542f48e8824e581fd0f89, fa66786862da49ae826fb519c7d89b5e
I20250627 13:52:24.382228 1524 raft_consensus.cc:2747] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 9f767210a91f44ae8b83b994131eee98 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250627 13:52:24.411013 1525 raft_consensus.cc:491] T fc3d1ee303e64241930642d9f13a3fd0 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:24.411228 1525 raft_consensus.cc:513] T fc3d1ee303e64241930642d9f13a3fd0 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:24.412734 1525 leader_election.cc:290] T fc3d1ee303e64241930642d9f13a3fd0 P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499), 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70:39887)
I20250627 13:52:24.433410 1254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc3d1ee303e64241930642d9f13a3fd0" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c28ff2fe9424379819d478fdd093cfc" is_pre_election: true
I20250627 13:52:24.433430 730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc3d1ee303e64241930642d9f13a3fd0" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c4f59060974a3b823305b9dd339a7e" is_pre_election: true
I20250627 13:52:24.433724 1254 raft_consensus.cc:2466] T fc3d1ee303e64241930642d9f13a3fd0 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 0.
I20250627 13:52:24.433722 730 raft_consensus.cc:2466] T fc3d1ee303e64241930642d9f13a3fd0 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 0.
I20250627 13:52:24.434345 928 leader_election.cc:304] T fc3d1ee303e64241930642d9f13a3fd0 P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 50e8c97891d74b38b5c130d3bcbabcb6, 5c28ff2fe9424379819d478fdd093cfc, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:24.434773 1525 raft_consensus.cc:2802] T fc3d1ee303e64241930642d9f13a3fd0 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:24.434911 1525 raft_consensus.cc:491] T fc3d1ee303e64241930642d9f13a3fd0 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:24.435016 1525 raft_consensus.cc:3058] T fc3d1ee303e64241930642d9f13a3fd0 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:24.437206 1525 raft_consensus.cc:513] T fc3d1ee303e64241930642d9f13a3fd0 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:24.437899 1525 leader_election.cc:290] T fc3d1ee303e64241930642d9f13a3fd0 P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 election: Requested vote from peers 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499), 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70:39887)
I20250627 13:52:24.438262 730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc3d1ee303e64241930642d9f13a3fd0" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c4f59060974a3b823305b9dd339a7e"
I20250627 13:52:24.438463 730 raft_consensus.cc:3058] T fc3d1ee303e64241930642d9f13a3fd0 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:24.438527 1254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc3d1ee303e64241930642d9f13a3fd0" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c28ff2fe9424379819d478fdd093cfc"
I20250627 13:52:24.438722 1254 raft_consensus.cc:3058] T fc3d1ee303e64241930642d9f13a3fd0 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:24.440402 730 raft_consensus.cc:2466] T fc3d1ee303e64241930642d9f13a3fd0 P 75c4f59060974a3b823305b9dd339a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 1.
I20250627 13:52:24.440562 1254 raft_consensus.cc:2466] T fc3d1ee303e64241930642d9f13a3fd0 P 5c28ff2fe9424379819d478fdd093cfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 1.
I20250627 13:52:24.440820 928 leader_election.cc:304] T fc3d1ee303e64241930642d9f13a3fd0 P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50e8c97891d74b38b5c130d3bcbabcb6, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:24.441092 1525 raft_consensus.cc:2802] T fc3d1ee303e64241930642d9f13a3fd0 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:24.441473 1525 raft_consensus.cc:695] T fc3d1ee303e64241930642d9f13a3fd0 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 LEADER]: Becoming Leader. State: Replica: 50e8c97891d74b38b5c130d3bcbabcb6, State: Running, Role: LEADER
I20250627 13:52:24.442211 1525 consensus_queue.cc:237] T fc3d1ee303e64241930642d9f13a3fd0 P 50e8c97891d74b38b5c130d3bcbabcb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:24.445783 447 catalog_manager.cc:5582] T fc3d1ee303e64241930642d9f13a3fd0 P 50e8c97891d74b38b5c130d3bcbabcb6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68). New cstate: current_term: 1 leader_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } health_report { overall_health: UNKNOWN } } }
I20250627 13:52:24.748528 1352 consensus_queue.cc:1035] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 957a8a8be3f542f48e8824e581fd0f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:24.756022 1352 consensus_queue.cc:1035] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 957a8a8be3f542f48e8824e581fd0f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:24.783742 1530 consensus_queue.cc:1035] T 5626734eb5a8437b97dd89f03c131ffa P fa66786862da49ae826fb519c7d89b5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:24.788930 597 raft_consensus.cc:3058] T 5626734eb5a8437b97dd89f03c131ffa P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:24.793239 1541 consensus_queue.cc:1035] T 5626734eb5a8437b97dd89f03c131ffa P fa66786862da49ae826fb519c7d89b5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250627 13:52:24.851939 1549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:24.852285 1549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:24.859864 1549 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250627 13:52:24.875809 1525 consensus_queue.cc:1035] T fc3d1ee303e64241930642d9f13a3fd0 P 50e8c97891d74b38b5c130d3bcbabcb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:24.882196 1547 consensus_queue.cc:1035] T fc3d1ee303e64241930642d9f13a3fd0 P 50e8c97891d74b38b5c130d3bcbabcb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 232.476 ms
time per row: 0.116238 ms
I20250627 13:52:25.240968 456 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49852:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250627 13:52:25.249272 1234 tablet_service.cc:1468] Processing CreateTablet for tablet f3721cf7c3a54f3e9e9cc88fc7ce7691 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b6687fc2ac1842ab96b1cef1ba60f65f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:25.249408 972 tablet_service.cc:1468] Processing CreateTablet for tablet f3721cf7c3a54f3e9e9cc88fc7ce7691 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b6687fc2ac1842ab96b1cef1ba60f65f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:25.249907 972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3721cf7c3a54f3e9e9cc88fc7ce7691. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:25.250324 971 tablet_service.cc:1468] Processing CreateTablet for tablet 148debd8ad384f51b509200b90ad4fdc (DEFAULT_TABLE table=rebalance_test_table_3 [id=b6687fc2ac1842ab96b1cef1ba60f65f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:25.250715 971 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 148debd8ad384f51b509200b90ad4fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:25.250917 710 tablet_service.cc:1468] Processing CreateTablet for tablet f3721cf7c3a54f3e9e9cc88fc7ce7691 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b6687fc2ac1842ab96b1cef1ba60f65f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:25.250941 709 tablet_service.cc:1468] Processing CreateTablet for tablet 148debd8ad384f51b509200b90ad4fdc (DEFAULT_TABLE table=rebalance_test_table_3 [id=b6687fc2ac1842ab96b1cef1ba60f65f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:25.251405 710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3721cf7c3a54f3e9e9cc88fc7ce7691. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:25.251636 709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 148debd8ad384f51b509200b90ad4fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:25.254225 1102 tablet_service.cc:1468] Processing CreateTablet for tablet 76a58b66489c4dd68d5c037aaffa9835 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b6687fc2ac1842ab96b1cef1ba60f65f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:25.254462 1233 tablet_service.cc:1468] Processing CreateTablet for tablet 148debd8ad384f51b509200b90ad4fdc (DEFAULT_TABLE table=rebalance_test_table_3 [id=b6687fc2ac1842ab96b1cef1ba60f65f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:25.254634 1102 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a58b66489c4dd68d5c037aaffa9835. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:25.254859 1233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 148debd8ad384f51b509200b90ad4fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:25.257951 1234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3721cf7c3a54f3e9e9cc88fc7ce7691. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:25.258205 841 tablet_service.cc:1468] Processing CreateTablet for tablet 76a58b66489c4dd68d5c037aaffa9835 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b6687fc2ac1842ab96b1cef1ba60f65f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:25.258608 841 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a58b66489c4dd68d5c037aaffa9835. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:25.262368 578 tablet_service.cc:1468] Processing CreateTablet for tablet 76a58b66489c4dd68d5c037aaffa9835 (DEFAULT_TABLE table=rebalance_test_table_3 [id=b6687fc2ac1842ab96b1cef1ba60f65f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:25.262799 578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a58b66489c4dd68d5c037aaffa9835. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:25.270313 1620 tablet_bootstrap.cc:492] T 148debd8ad384f51b509200b90ad4fdc P 75c4f59060974a3b823305b9dd339a7e: Bootstrap starting.
I20250627 13:52:25.271673 1620 tablet_bootstrap.cc:654] T 148debd8ad384f51b509200b90ad4fdc P 75c4f59060974a3b823305b9dd339a7e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:25.272727 1621 tablet_bootstrap.cc:492] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 5c28ff2fe9424379819d478fdd093cfc: Bootstrap starting.
I20250627 13:52:25.273319 1624 tablet_bootstrap.cc:492] T 76a58b66489c4dd68d5c037aaffa9835 P fa66786862da49ae826fb519c7d89b5e: Bootstrap starting.
I20250627 13:52:25.274657 1621 tablet_bootstrap.cc:654] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 5c28ff2fe9424379819d478fdd093cfc: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:25.275360 1624 tablet_bootstrap.cc:654] T 76a58b66489c4dd68d5c037aaffa9835 P fa66786862da49ae826fb519c7d89b5e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:25.275769 1620 tablet_bootstrap.cc:492] T 148debd8ad384f51b509200b90ad4fdc P 75c4f59060974a3b823305b9dd339a7e: No bootstrap required, opened a new log
I20250627 13:52:25.275962 1620 ts_tablet_manager.cc:1397] T 148debd8ad384f51b509200b90ad4fdc P 75c4f59060974a3b823305b9dd339a7e: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250627 13:52:25.276723 1623 tablet_bootstrap.cc:492] T 76a58b66489c4dd68d5c037aaffa9835 P 9f767210a91f44ae8b83b994131eee98: Bootstrap starting.
I20250627 13:52:25.276960 1620 raft_consensus.cc:357] T 148debd8ad384f51b509200b90ad4fdc P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:25.277154 1620 raft_consensus.cc:383] T 148debd8ad384f51b509200b90ad4fdc P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:25.277248 1620 raft_consensus.cc:738] T 148debd8ad384f51b509200b90ad4fdc P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c4f59060974a3b823305b9dd339a7e, State: Initialized, Role: FOLLOWER
I20250627 13:52:25.277472 1620 consensus_queue.cc:260] T 148debd8ad384f51b509200b90ad4fdc P 75c4f59060974a3b823305b9dd339a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:25.278105 1620 ts_tablet_manager.cc:1428] T 148debd8ad384f51b509200b90ad4fdc P 75c4f59060974a3b823305b9dd339a7e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:52:25.278461 1620 tablet_bootstrap.cc:492] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 75c4f59060974a3b823305b9dd339a7e: Bootstrap starting.
I20250627 13:52:25.278659 1623 tablet_bootstrap.cc:654] T 76a58b66489c4dd68d5c037aaffa9835 P 9f767210a91f44ae8b83b994131eee98: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:25.280308 1620 tablet_bootstrap.cc:654] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 75c4f59060974a3b823305b9dd339a7e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:25.282948 1627 tablet_bootstrap.cc:492] T 148debd8ad384f51b509200b90ad4fdc P 50e8c97891d74b38b5c130d3bcbabcb6: Bootstrap starting.
I20250627 13:52:25.284991 1627 tablet_bootstrap.cc:654] T 148debd8ad384f51b509200b90ad4fdc P 50e8c97891d74b38b5c130d3bcbabcb6: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:25.286082 1624 tablet_bootstrap.cc:492] T 76a58b66489c4dd68d5c037aaffa9835 P fa66786862da49ae826fb519c7d89b5e: No bootstrap required, opened a new log
I20250627 13:52:25.286316 1624 ts_tablet_manager.cc:1397] T 76a58b66489c4dd68d5c037aaffa9835 P fa66786862da49ae826fb519c7d89b5e: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.006s
I20250627 13:52:25.287381 1624 raft_consensus.cc:357] T 76a58b66489c4dd68d5c037aaffa9835 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } }
I20250627 13:52:25.287714 1624 raft_consensus.cc:383] T 76a58b66489c4dd68d5c037aaffa9835 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:25.287865 1624 raft_consensus.cc:738] T 76a58b66489c4dd68d5c037aaffa9835 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa66786862da49ae826fb519c7d89b5e, State: Initialized, Role: FOLLOWER
I20250627 13:52:25.288235 1624 consensus_queue.cc:260] T 76a58b66489c4dd68d5c037aaffa9835 P fa66786862da49ae826fb519c7d89b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } }
I20250627 13:52:25.289091 1624 ts_tablet_manager.cc:1428] T 76a58b66489c4dd68d5c037aaffa9835 P fa66786862da49ae826fb519c7d89b5e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250627 13:52:25.290692 1623 tablet_bootstrap.cc:492] T 76a58b66489c4dd68d5c037aaffa9835 P 9f767210a91f44ae8b83b994131eee98: No bootstrap required, opened a new log
I20250627 13:52:25.290915 1623 ts_tablet_manager.cc:1397] T 76a58b66489c4dd68d5c037aaffa9835 P 9f767210a91f44ae8b83b994131eee98: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20250627 13:52:25.291532 1627 tablet_bootstrap.cc:492] T 148debd8ad384f51b509200b90ad4fdc P 50e8c97891d74b38b5c130d3bcbabcb6: No bootstrap required, opened a new log
I20250627 13:52:25.291731 1627 ts_tablet_manager.cc:1397] T 148debd8ad384f51b509200b90ad4fdc P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20250627 13:52:25.292164 1621 tablet_bootstrap.cc:492] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 5c28ff2fe9424379819d478fdd093cfc: No bootstrap required, opened a new log
I20250627 13:52:25.292191 1623 raft_consensus.cc:357] T 76a58b66489c4dd68d5c037aaffa9835 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } }
I20250627 13:52:25.292363 1621 ts_tablet_manager.cc:1397] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 5c28ff2fe9424379819d478fdd093cfc: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.004s
I20250627 13:52:25.292411 1623 raft_consensus.cc:383] T 76a58b66489c4dd68d5c037aaffa9835 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:25.292537 1623 raft_consensus.cc:738] T 76a58b66489c4dd68d5c037aaffa9835 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f767210a91f44ae8b83b994131eee98, State: Initialized, Role: FOLLOWER
I20250627 13:52:25.292510 1627 raft_consensus.cc:357] T 148debd8ad384f51b509200b90ad4fdc P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:25.292685 1627 raft_consensus.cc:383] T 148debd8ad384f51b509200b90ad4fdc P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:25.292779 1627 raft_consensus.cc:738] T 148debd8ad384f51b509200b90ad4fdc P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e8c97891d74b38b5c130d3bcbabcb6, State: Initialized, Role: FOLLOWER
I20250627 13:52:25.292995 1623 consensus_queue.cc:260] T 76a58b66489c4dd68d5c037aaffa9835 P 9f767210a91f44ae8b83b994131eee98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } }
I20250627 13:52:25.293020 1627 consensus_queue.cc:260] T 148debd8ad384f51b509200b90ad4fdc P 50e8c97891d74b38b5c130d3bcbabcb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:25.293174 1621 raft_consensus.cc:357] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:25.293545 1621 raft_consensus.cc:383] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:25.293699 1627 ts_tablet_manager.cc:1428] T 148debd8ad384f51b509200b90ad4fdc P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250627 13:52:25.293680 1621 raft_consensus.cc:738] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c28ff2fe9424379819d478fdd093cfc, State: Initialized, Role: FOLLOWER
I20250627 13:52:25.293934 1621 consensus_queue.cc:260] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 5c28ff2fe9424379819d478fdd093cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:25.294545 1621 ts_tablet_manager.cc:1428] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 5c28ff2fe9424379819d478fdd093cfc: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250627 13:52:25.296180 1623 ts_tablet_manager.cc:1428] T 76a58b66489c4dd68d5c037aaffa9835 P 9f767210a91f44ae8b83b994131eee98: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250627 13:52:25.296751 1621 tablet_bootstrap.cc:492] T 148debd8ad384f51b509200b90ad4fdc P 5c28ff2fe9424379819d478fdd093cfc: Bootstrap starting.
I20250627 13:52:25.299039 1621 tablet_bootstrap.cc:654] T 148debd8ad384f51b509200b90ad4fdc P 5c28ff2fe9424379819d478fdd093cfc: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:25.302400 1621 tablet_bootstrap.cc:492] T 148debd8ad384f51b509200b90ad4fdc P 5c28ff2fe9424379819d478fdd093cfc: No bootstrap required, opened a new log
I20250627 13:52:25.302865 1621 ts_tablet_manager.cc:1397] T 148debd8ad384f51b509200b90ad4fdc P 5c28ff2fe9424379819d478fdd093cfc: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20250627 13:52:25.303333 1632 tablet_bootstrap.cc:492] T 76a58b66489c4dd68d5c037aaffa9835 P 957a8a8be3f542f48e8824e581fd0f89: Bootstrap starting.
I20250627 13:52:25.303900 1621 raft_consensus.cc:357] T 148debd8ad384f51b509200b90ad4fdc P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:25.304215 1621 raft_consensus.cc:383] T 148debd8ad384f51b509200b90ad4fdc P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:25.304327 1621 raft_consensus.cc:738] T 148debd8ad384f51b509200b90ad4fdc P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c28ff2fe9424379819d478fdd093cfc, State: Initialized, Role: FOLLOWER
I20250627 13:52:25.304701 1621 consensus_queue.cc:260] T 148debd8ad384f51b509200b90ad4fdc P 5c28ff2fe9424379819d478fdd093cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:25.305629 1621 ts_tablet_manager.cc:1428] T 148debd8ad384f51b509200b90ad4fdc P 5c28ff2fe9424379819d478fdd093cfc: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250627 13:52:25.306726 1632 tablet_bootstrap.cc:654] T 76a58b66489c4dd68d5c037aaffa9835 P 957a8a8be3f542f48e8824e581fd0f89: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:25.310223 1627 tablet_bootstrap.cc:492] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 50e8c97891d74b38b5c130d3bcbabcb6: Bootstrap starting.
I20250627 13:52:25.312633 1627 tablet_bootstrap.cc:654] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 50e8c97891d74b38b5c130d3bcbabcb6: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:25.315619 1632 tablet_bootstrap.cc:492] T 76a58b66489c4dd68d5c037aaffa9835 P 957a8a8be3f542f48e8824e581fd0f89: No bootstrap required, opened a new log
I20250627 13:52:25.315779 1627 tablet_bootstrap.cc:492] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 50e8c97891d74b38b5c130d3bcbabcb6: No bootstrap required, opened a new log
I20250627 13:52:25.315821 1632 ts_tablet_manager.cc:1397] T 76a58b66489c4dd68d5c037aaffa9835 P 957a8a8be3f542f48e8824e581fd0f89: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20250627 13:52:25.315989 1627 ts_tablet_manager.cc:1397] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20250627 13:52:25.316668 1632 raft_consensus.cc:357] T 76a58b66489c4dd68d5c037aaffa9835 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } }
I20250627 13:52:25.316917 1632 raft_consensus.cc:383] T 76a58b66489c4dd68d5c037aaffa9835 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:25.317058 1632 raft_consensus.cc:738] T 76a58b66489c4dd68d5c037aaffa9835 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957a8a8be3f542f48e8824e581fd0f89, State: Initialized, Role: FOLLOWER
I20250627 13:52:25.316901 1627 raft_consensus.cc:357] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:25.317169 1627 raft_consensus.cc:383] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:25.317267 1627 raft_consensus.cc:738] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e8c97891d74b38b5c130d3bcbabcb6, State: Initialized, Role: FOLLOWER
I20250627 13:52:25.317411 1632 consensus_queue.cc:260] T 76a58b66489c4dd68d5c037aaffa9835 P 957a8a8be3f542f48e8824e581fd0f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } }
I20250627 13:52:25.317580 1627 consensus_queue.cc:260] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 50e8c97891d74b38b5c130d3bcbabcb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:25.318249 1627 ts_tablet_manager.cc:1428] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250627 13:52:25.318720 1525 raft_consensus.cc:491] T 148debd8ad384f51b509200b90ad4fdc P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:25.318858 1525 raft_consensus.cc:513] T 148debd8ad384f51b509200b90ad4fdc P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:25.319494 1525 leader_election.cc:290] T 148debd8ad384f51b509200b90ad4fdc P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499), 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70:39887)
I20250627 13:52:25.320493 730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148debd8ad384f51b509200b90ad4fdc" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c4f59060974a3b823305b9dd339a7e" is_pre_election: true
I20250627 13:52:25.320721 730 raft_consensus.cc:2466] T 148debd8ad384f51b509200b90ad4fdc P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 0.
I20250627 13:52:25.321050 1254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148debd8ad384f51b509200b90ad4fdc" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c28ff2fe9424379819d478fdd093cfc" is_pre_election: true
I20250627 13:52:25.321233 1254 raft_consensus.cc:2466] T 148debd8ad384f51b509200b90ad4fdc P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 0.
I20250627 13:52:25.322014 926 leader_election.cc:304] T 148debd8ad384f51b509200b90ad4fdc P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50e8c97891d74b38b5c130d3bcbabcb6, 5c28ff2fe9424379819d478fdd093cfc; no voters:
I20250627 13:52:25.322531 1525 raft_consensus.cc:2802] T 148debd8ad384f51b509200b90ad4fdc P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:25.322547 1632 ts_tablet_manager.cc:1428] T 76a58b66489c4dd68d5c037aaffa9835 P 957a8a8be3f542f48e8824e581fd0f89: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250627 13:52:25.322640 1525 raft_consensus.cc:491] T 148debd8ad384f51b509200b90ad4fdc P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:25.322883 1525 raft_consensus.cc:3058] T 148debd8ad384f51b509200b90ad4fdc P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:25.324445 1620 tablet_bootstrap.cc:492] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 75c4f59060974a3b823305b9dd339a7e: No bootstrap required, opened a new log
I20250627 13:52:25.324682 1620 ts_tablet_manager.cc:1397] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 75c4f59060974a3b823305b9dd339a7e: Time spent bootstrapping tablet: real 0.046s user 0.011s sys 0.016s
I20250627 13:52:25.325502 1620 raft_consensus.cc:357] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:25.325572 1525 raft_consensus.cc:513] T 148debd8ad384f51b509200b90ad4fdc P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:25.325776 1620 raft_consensus.cc:383] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:25.325865 1620 raft_consensus.cc:738] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c4f59060974a3b823305b9dd339a7e, State: Initialized, Role: FOLLOWER
I20250627 13:52:25.326109 1620 consensus_queue.cc:260] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 75c4f59060974a3b823305b9dd339a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:25.326568 1620 ts_tablet_manager.cc:1428] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 75c4f59060974a3b823305b9dd339a7e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:52:25.327870 1525 leader_election.cc:290] T 148debd8ad384f51b509200b90ad4fdc P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 election: Requested vote from peers 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499), 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70:39887)
I20250627 13:52:25.329502 1254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148debd8ad384f51b509200b90ad4fdc" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c28ff2fe9424379819d478fdd093cfc"
I20250627 13:52:25.329696 1254 raft_consensus.cc:3058] T 148debd8ad384f51b509200b90ad4fdc P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:25.329639 730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "148debd8ad384f51b509200b90ad4fdc" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c4f59060974a3b823305b9dd339a7e"
I20250627 13:52:25.329849 730 raft_consensus.cc:3058] T 148debd8ad384f51b509200b90ad4fdc P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:25.332149 730 raft_consensus.cc:2466] T 148debd8ad384f51b509200b90ad4fdc P 75c4f59060974a3b823305b9dd339a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 1.
I20250627 13:52:25.332546 1254 raft_consensus.cc:2466] T 148debd8ad384f51b509200b90ad4fdc P 5c28ff2fe9424379819d478fdd093cfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 1.
I20250627 13:52:25.332576 928 leader_election.cc:304] T 148debd8ad384f51b509200b90ad4fdc P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50e8c97891d74b38b5c130d3bcbabcb6, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:25.332913 1525 raft_consensus.cc:2802] T 148debd8ad384f51b509200b90ad4fdc P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:25.333048 1525 raft_consensus.cc:695] T 148debd8ad384f51b509200b90ad4fdc P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 LEADER]: Becoming Leader. State: Replica: 50e8c97891d74b38b5c130d3bcbabcb6, State: Running, Role: LEADER
I20250627 13:52:25.333273 1525 consensus_queue.cc:237] T 148debd8ad384f51b509200b90ad4fdc P 50e8c97891d74b38b5c130d3bcbabcb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:25.339421 455 catalog_manager.cc:5582] T 148debd8ad384f51b509200b90ad4fdc P 50e8c97891d74b38b5c130d3bcbabcb6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68). New cstate: current_term: 1 leader_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } health_report { overall_health: UNKNOWN } } }
I20250627 13:52:25.396579 1538 raft_consensus.cc:491] T 76a58b66489c4dd68d5c037aaffa9835 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:25.396765 1538 raft_consensus.cc:513] T 76a58b66489c4dd68d5c037aaffa9835 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } }
I20250627 13:52:25.397433 1538 leader_election.cc:290] T 76a58b66489c4dd68d5c037aaffa9835 P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023), fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747)
I20250627 13:52:25.397907 598 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76a58b66489c4dd68d5c037aaffa9835" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98" is_pre_election: true
I20250627 13:52:25.398011 861 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76a58b66489c4dd68d5c037aaffa9835" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e" is_pre_election: true
I20250627 13:52:25.398116 598 raft_consensus.cc:2466] T 76a58b66489c4dd68d5c037aaffa9835 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 0.
I20250627 13:52:25.398245 861 raft_consensus.cc:2466] T 76a58b66489c4dd68d5c037aaffa9835 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 0.
I20250627 13:52:25.398536 1056 leader_election.cc:304] T 76a58b66489c4dd68d5c037aaffa9835 P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, 9f767210a91f44ae8b83b994131eee98; no voters:
I20250627 13:52:25.398785 1538 raft_consensus.cc:2802] T 76a58b66489c4dd68d5c037aaffa9835 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:25.398890 1538 raft_consensus.cc:491] T 76a58b66489c4dd68d5c037aaffa9835 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:25.398988 1538 raft_consensus.cc:3058] T 76a58b66489c4dd68d5c037aaffa9835 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:25.400893 1538 raft_consensus.cc:513] T 76a58b66489c4dd68d5c037aaffa9835 P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } }
I20250627 13:52:25.401432 1538 leader_election.cc:290] T 76a58b66489c4dd68d5c037aaffa9835 P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 election: Requested vote from peers 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023), fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747)
I20250627 13:52:25.401789 598 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76a58b66489c4dd68d5c037aaffa9835" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98"
I20250627 13:52:25.401867 861 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76a58b66489c4dd68d5c037aaffa9835" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e"
I20250627 13:52:25.401995 598 raft_consensus.cc:3058] T 76a58b66489c4dd68d5c037aaffa9835 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:25.402073 861 raft_consensus.cc:3058] T 76a58b66489c4dd68d5c037aaffa9835 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:25.403889 598 raft_consensus.cc:2466] T 76a58b66489c4dd68d5c037aaffa9835 P 9f767210a91f44ae8b83b994131eee98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 1.
I20250627 13:52:25.403920 861 raft_consensus.cc:2466] T 76a58b66489c4dd68d5c037aaffa9835 P fa66786862da49ae826fb519c7d89b5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 1.
I20250627 13:52:25.404349 1056 leader_election.cc:304] T 76a58b66489c4dd68d5c037aaffa9835 P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, 9f767210a91f44ae8b83b994131eee98, fa66786862da49ae826fb519c7d89b5e; no voters:
I20250627 13:52:25.404654 1538 raft_consensus.cc:2802] T 76a58b66489c4dd68d5c037aaffa9835 P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:25.404795 1538 raft_consensus.cc:695] T 76a58b66489c4dd68d5c037aaffa9835 P 957a8a8be3f542f48e8824e581fd0f89 [term 1 LEADER]: Becoming Leader. State: Replica: 957a8a8be3f542f48e8824e581fd0f89, State: Running, Role: LEADER
I20250627 13:52:25.405084 1538 consensus_queue.cc:237] T 76a58b66489c4dd68d5c037aaffa9835 P 957a8a8be3f542f48e8824e581fd0f89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } }
I20250627 13:52:25.407750 455 catalog_manager.cc:5582] T 76a58b66489c4dd68d5c037aaffa9835 P 957a8a8be3f542f48e8824e581fd0f89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69). New cstate: current_term: 1 leader_uuid: "957a8a8be3f542f48e8824e581fd0f89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } health_report { overall_health: UNKNOWN } } }
I20250627 13:52:25.486935 1525 raft_consensus.cc:491] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:25.487159 1525 raft_consensus.cc:513] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:25.487716 1525 leader_election.cc:290] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70:39887), 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499)
I20250627 13:52:25.488251 1254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3721cf7c3a54f3e9e9cc88fc7ce7691" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c28ff2fe9424379819d478fdd093cfc" is_pre_election: true
I20250627 13:52:25.488276 730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3721cf7c3a54f3e9e9cc88fc7ce7691" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c4f59060974a3b823305b9dd339a7e" is_pre_election: true
I20250627 13:52:25.488503 1254 raft_consensus.cc:2466] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 0.
I20250627 13:52:25.488507 730 raft_consensus.cc:2466] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 0.
I20250627 13:52:25.488993 926 leader_election.cc:304] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 50e8c97891d74b38b5c130d3bcbabcb6, 5c28ff2fe9424379819d478fdd093cfc, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:25.489332 1525 raft_consensus.cc:2802] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:25.489415 1525 raft_consensus.cc:491] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:25.489511 1525 raft_consensus.cc:3058] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:25.491402 1525 raft_consensus.cc:513] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:25.491940 1525 leader_election.cc:290] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 election: Requested vote from peers 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70:39887), 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499)
I20250627 13:52:25.492461 1254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3721cf7c3a54f3e9e9cc88fc7ce7691" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c28ff2fe9424379819d478fdd093cfc"
I20250627 13:52:25.492483 730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3721cf7c3a54f3e9e9cc88fc7ce7691" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c4f59060974a3b823305b9dd339a7e"
I20250627 13:52:25.492664 730 raft_consensus.cc:3058] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:25.492676 1254 raft_consensus.cc:3058] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:25.495661 730 raft_consensus.cc:2466] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 75c4f59060974a3b823305b9dd339a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 1.
I20250627 13:52:25.495645 1254 raft_consensus.cc:2466] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 5c28ff2fe9424379819d478fdd093cfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 1.
I20250627 13:52:25.496080 928 leader_election.cc:304] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 50e8c97891d74b38b5c130d3bcbabcb6, 5c28ff2fe9424379819d478fdd093cfc, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:25.496349 1525 raft_consensus.cc:2802] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:25.496507 1525 raft_consensus.cc:695] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 LEADER]: Becoming Leader. State: Replica: 50e8c97891d74b38b5c130d3bcbabcb6, State: Running, Role: LEADER
I20250627 13:52:25.496753 1525 consensus_queue.cc:237] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 50e8c97891d74b38b5c130d3bcbabcb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:25.499367 455 catalog_manager.cc:5582] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 50e8c97891d74b38b5c130d3bcbabcb6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68). New cstate: current_term: 1 leader_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } health_report { overall_health: UNKNOWN } } }
W20250627 13:52:25.827891 1638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:25.828248 1638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:25.836596 1638 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20250627 13:52:25.898277 1538 consensus_queue.cc:1035] T 76a58b66489c4dd68d5c037aaffa9835 P 957a8a8be3f542f48e8824e581fd0f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:25.904857 1525 consensus_queue.cc:1035] T 148debd8ad384f51b509200b90ad4fdc P 50e8c97891d74b38b5c130d3bcbabcb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:25.909474 1352 consensus_queue.cc:1035] T 76a58b66489c4dd68d5c037aaffa9835 P 957a8a8be3f542f48e8824e581fd0f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:25.946147 1547 consensus_queue.cc:1035] T 148debd8ad384f51b509200b90ad4fdc P 50e8c97891d74b38b5c130d3bcbabcb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:25.951411 1547 consensus_queue.cc:1035] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 50e8c97891d74b38b5c130d3bcbabcb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:25.971062 1547 consensus_queue.cc:1035] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 50e8c97891d74b38b5c130d3bcbabcb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 267.129 ms
time per row: 0.133564 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20250627 13:52:27.245110 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49852:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250627 13:52:27.254356 578 tablet_service.cc:1468] Processing CreateTablet for tablet a0d7bf2e65de471b9867ceb7430a20d3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=997d45aa93f5413a85c9a40b54ec5b3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:27.254506 841 tablet_service.cc:1468] Processing CreateTablet for tablet a0d7bf2e65de471b9867ceb7430a20d3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=997d45aa93f5413a85c9a40b54ec5b3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:27.254874 841 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0d7bf2e65de471b9867ceb7430a20d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:27.254873 578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0d7bf2e65de471b9867ceb7430a20d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:27.255604 1234 tablet_service.cc:1468] Processing CreateTablet for tablet f55615149da94dd1be6f5f09c4a2f026 (DEFAULT_TABLE table=rebalance_test_table_4 [id=997d45aa93f5413a85c9a40b54ec5b3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:27.255913 1234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f55615149da94dd1be6f5f09c4a2f026. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:27.258164 1102 tablet_service.cc:1468] Processing CreateTablet for tablet a0d7bf2e65de471b9867ceb7430a20d3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=997d45aa93f5413a85c9a40b54ec5b3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:27.259306 1102 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0d7bf2e65de471b9867ceb7430a20d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:27.259183 972 tablet_service.cc:1468] Processing CreateTablet for tablet f55615149da94dd1be6f5f09c4a2f026 (DEFAULT_TABLE table=rebalance_test_table_4 [id=997d45aa93f5413a85c9a40b54ec5b3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:27.259752 972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f55615149da94dd1be6f5f09c4a2f026. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:27.261758 709 tablet_service.cc:1468] Processing CreateTablet for tablet f55615149da94dd1be6f5f09c4a2f026 (DEFAULT_TABLE table=rebalance_test_table_4 [id=997d45aa93f5413a85c9a40b54ec5b3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:27.262178 709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f55615149da94dd1be6f5f09c4a2f026. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:27.266229 840 tablet_service.cc:1468] Processing CreateTablet for tablet 383a7d030ab5416abcc36eab73600269 (DEFAULT_TABLE table=rebalance_test_table_4 [id=997d45aa93f5413a85c9a40b54ec5b3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:27.266604 840 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 383a7d030ab5416abcc36eab73600269. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:27.269747 577 tablet_service.cc:1468] Processing CreateTablet for tablet 383a7d030ab5416abcc36eab73600269 (DEFAULT_TABLE table=rebalance_test_table_4 [id=997d45aa93f5413a85c9a40b54ec5b3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:27.270069 1103 tablet_service.cc:1468] Processing CreateTablet for tablet 383a7d030ab5416abcc36eab73600269 (DEFAULT_TABLE table=rebalance_test_table_4 [id=997d45aa93f5413a85c9a40b54ec5b3c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:27.270258 577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 383a7d030ab5416abcc36eab73600269. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:27.270481 1103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 383a7d030ab5416abcc36eab73600269. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:27.282056 1718 tablet_bootstrap.cc:492] T f55615149da94dd1be6f5f09c4a2f026 P 50e8c97891d74b38b5c130d3bcbabcb6: Bootstrap starting.
I20250627 13:52:27.284459 1718 tablet_bootstrap.cc:654] T f55615149da94dd1be6f5f09c4a2f026 P 50e8c97891d74b38b5c130d3bcbabcb6: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:27.284629 1717 tablet_bootstrap.cc:492] T f55615149da94dd1be6f5f09c4a2f026 P 5c28ff2fe9424379819d478fdd093cfc: Bootstrap starting.
I20250627 13:52:27.285068 1714 tablet_bootstrap.cc:492] T a0d7bf2e65de471b9867ceb7430a20d3 P 9f767210a91f44ae8b83b994131eee98: Bootstrap starting.
I20250627 13:52:27.286489 1719 tablet_bootstrap.cc:492] T f55615149da94dd1be6f5f09c4a2f026 P 75c4f59060974a3b823305b9dd339a7e: Bootstrap starting.
I20250627 13:52:27.286823 1714 tablet_bootstrap.cc:654] T a0d7bf2e65de471b9867ceb7430a20d3 P 9f767210a91f44ae8b83b994131eee98: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:27.288182 1717 tablet_bootstrap.cc:654] T f55615149da94dd1be6f5f09c4a2f026 P 5c28ff2fe9424379819d478fdd093cfc: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:27.288700 1713 tablet_bootstrap.cc:492] T a0d7bf2e65de471b9867ceb7430a20d3 P fa66786862da49ae826fb519c7d89b5e: Bootstrap starting.
I20250627 13:52:27.291954 1719 tablet_bootstrap.cc:654] T f55615149da94dd1be6f5f09c4a2f026 P 75c4f59060974a3b823305b9dd339a7e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:27.293588 1713 tablet_bootstrap.cc:654] T a0d7bf2e65de471b9867ceb7430a20d3 P fa66786862da49ae826fb519c7d89b5e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:27.293740 1716 tablet_bootstrap.cc:492] T a0d7bf2e65de471b9867ceb7430a20d3 P 957a8a8be3f542f48e8824e581fd0f89: Bootstrap starting.
I20250627 13:52:27.295825 1716 tablet_bootstrap.cc:654] T a0d7bf2e65de471b9867ceb7430a20d3 P 957a8a8be3f542f48e8824e581fd0f89: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:27.295926 1713 tablet_bootstrap.cc:492] T a0d7bf2e65de471b9867ceb7430a20d3 P fa66786862da49ae826fb519c7d89b5e: No bootstrap required, opened a new log
I20250627 13:52:27.296104 1713 ts_tablet_manager.cc:1397] T a0d7bf2e65de471b9867ceb7430a20d3 P fa66786862da49ae826fb519c7d89b5e: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20250627 13:52:27.297107 1713 raft_consensus.cc:357] T a0d7bf2e65de471b9867ceb7430a20d3 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:27.298243 1713 raft_consensus.cc:383] T a0d7bf2e65de471b9867ceb7430a20d3 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:27.298367 1713 raft_consensus.cc:738] T a0d7bf2e65de471b9867ceb7430a20d3 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa66786862da49ae826fb519c7d89b5e, State: Initialized, Role: FOLLOWER
I20250627 13:52:27.298629 1714 tablet_bootstrap.cc:492] T a0d7bf2e65de471b9867ceb7430a20d3 P 9f767210a91f44ae8b83b994131eee98: No bootstrap required, opened a new log
I20250627 13:52:27.298823 1714 ts_tablet_manager.cc:1397] T a0d7bf2e65de471b9867ceb7430a20d3 P 9f767210a91f44ae8b83b994131eee98: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.005s
I20250627 13:52:27.298961 1713 consensus_queue.cc:260] T a0d7bf2e65de471b9867ceb7430a20d3 P fa66786862da49ae826fb519c7d89b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:27.299520 1713 ts_tablet_manager.cc:1428] T a0d7bf2e65de471b9867ceb7430a20d3 P fa66786862da49ae826fb519c7d89b5e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250627 13:52:27.299664 1714 raft_consensus.cc:357] T a0d7bf2e65de471b9867ceb7430a20d3 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:27.299894 1714 raft_consensus.cc:383] T a0d7bf2e65de471b9867ceb7430a20d3 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:27.300005 1714 raft_consensus.cc:738] T a0d7bf2e65de471b9867ceb7430a20d3 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f767210a91f44ae8b83b994131eee98, State: Initialized, Role: FOLLOWER
I20250627 13:52:27.300246 1714 consensus_queue.cc:260] T a0d7bf2e65de471b9867ceb7430a20d3 P 9f767210a91f44ae8b83b994131eee98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:27.300352 1713 tablet_bootstrap.cc:492] T 383a7d030ab5416abcc36eab73600269 P fa66786862da49ae826fb519c7d89b5e: Bootstrap starting.
I20250627 13:52:27.301522 1718 tablet_bootstrap.cc:492] T f55615149da94dd1be6f5f09c4a2f026 P 50e8c97891d74b38b5c130d3bcbabcb6: No bootstrap required, opened a new log
I20250627 13:52:27.301739 1718 ts_tablet_manager.cc:1397] T f55615149da94dd1be6f5f09c4a2f026 P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.007s
I20250627 13:52:27.301885 1713 tablet_bootstrap.cc:654] T 383a7d030ab5416abcc36eab73600269 P fa66786862da49ae826fb519c7d89b5e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:27.302502 1719 tablet_bootstrap.cc:492] T f55615149da94dd1be6f5f09c4a2f026 P 75c4f59060974a3b823305b9dd339a7e: No bootstrap required, opened a new log
I20250627 13:52:27.302695 1719 ts_tablet_manager.cc:1397] T f55615149da94dd1be6f5f09c4a2f026 P 75c4f59060974a3b823305b9dd339a7e: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20250627 13:52:27.302647 1718 raft_consensus.cc:357] T f55615149da94dd1be6f5f09c4a2f026 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:27.302908 1718 raft_consensus.cc:383] T f55615149da94dd1be6f5f09c4a2f026 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:27.303082 1718 raft_consensus.cc:738] T f55615149da94dd1be6f5f09c4a2f026 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e8c97891d74b38b5c130d3bcbabcb6, State: Initialized, Role: FOLLOWER
I20250627 13:52:27.303400 1718 consensus_queue.cc:260] T f55615149da94dd1be6f5f09c4a2f026 P 50e8c97891d74b38b5c130d3bcbabcb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:27.303506 1719 raft_consensus.cc:357] T f55615149da94dd1be6f5f09c4a2f026 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:27.303794 1719 raft_consensus.cc:383] T f55615149da94dd1be6f5f09c4a2f026 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:27.303951 1719 raft_consensus.cc:738] T f55615149da94dd1be6f5f09c4a2f026 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c4f59060974a3b823305b9dd339a7e, State: Initialized, Role: FOLLOWER
I20250627 13:52:27.304117 1718 ts_tablet_manager.cc:1428] T f55615149da94dd1be6f5f09c4a2f026 P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:52:27.304276 1719 consensus_queue.cc:260] T f55615149da94dd1be6f5f09c4a2f026 P 75c4f59060974a3b823305b9dd339a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:27.305033 1719 ts_tablet_manager.cc:1428] T f55615149da94dd1be6f5f09c4a2f026 P 75c4f59060974a3b823305b9dd339a7e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:52:27.311182 1716 tablet_bootstrap.cc:492] T a0d7bf2e65de471b9867ceb7430a20d3 P 957a8a8be3f542f48e8824e581fd0f89: No bootstrap required, opened a new log
I20250627 13:52:27.311486 1716 ts_tablet_manager.cc:1397] T a0d7bf2e65de471b9867ceb7430a20d3 P 957a8a8be3f542f48e8824e581fd0f89: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20250627 13:52:27.311669 1714 ts_tablet_manager.cc:1428] T a0d7bf2e65de471b9867ceb7430a20d3 P 9f767210a91f44ae8b83b994131eee98: Time spent starting tablet: real 0.013s user 0.001s sys 0.008s
I20250627 13:52:27.312816 1714 tablet_bootstrap.cc:492] T 383a7d030ab5416abcc36eab73600269 P 9f767210a91f44ae8b83b994131eee98: Bootstrap starting.
I20250627 13:52:27.312923 1716 raft_consensus.cc:357] T a0d7bf2e65de471b9867ceb7430a20d3 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:27.313243 1716 raft_consensus.cc:383] T a0d7bf2e65de471b9867ceb7430a20d3 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:27.313371 1716 raft_consensus.cc:738] T a0d7bf2e65de471b9867ceb7430a20d3 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957a8a8be3f542f48e8824e581fd0f89, State: Initialized, Role: FOLLOWER
I20250627 13:52:27.313692 1716 consensus_queue.cc:260] T a0d7bf2e65de471b9867ceb7430a20d3 P 957a8a8be3f542f48e8824e581fd0f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:27.314546 1716 ts_tablet_manager.cc:1428] T a0d7bf2e65de471b9867ceb7430a20d3 P 957a8a8be3f542f48e8824e581fd0f89: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250627 13:52:27.314935 1716 tablet_bootstrap.cc:492] T 383a7d030ab5416abcc36eab73600269 P 957a8a8be3f542f48e8824e581fd0f89: Bootstrap starting.
I20250627 13:52:27.315590 1714 tablet_bootstrap.cc:654] T 383a7d030ab5416abcc36eab73600269 P 9f767210a91f44ae8b83b994131eee98: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:27.317265 1716 tablet_bootstrap.cc:654] T 383a7d030ab5416abcc36eab73600269 P 957a8a8be3f542f48e8824e581fd0f89: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:27.320978 1714 tablet_bootstrap.cc:492] T 383a7d030ab5416abcc36eab73600269 P 9f767210a91f44ae8b83b994131eee98: No bootstrap required, opened a new log
I20250627 13:52:27.321076 1717 tablet_bootstrap.cc:492] T f55615149da94dd1be6f5f09c4a2f026 P 5c28ff2fe9424379819d478fdd093cfc: No bootstrap required, opened a new log
I20250627 13:52:27.321206 1714 ts_tablet_manager.cc:1397] T 383a7d030ab5416abcc36eab73600269 P 9f767210a91f44ae8b83b994131eee98: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20250627 13:52:27.321302 1717 ts_tablet_manager.cc:1397] T f55615149da94dd1be6f5f09c4a2f026 P 5c28ff2fe9424379819d478fdd093cfc: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.004s
I20250627 13:52:27.322103 1714 raft_consensus.cc:357] T 383a7d030ab5416abcc36eab73600269 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:27.322114 1717 raft_consensus.cc:357] T f55615149da94dd1be6f5f09c4a2f026 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:27.322305 1717 raft_consensus.cc:383] T f55615149da94dd1be6f5f09c4a2f026 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:27.322407 1717 raft_consensus.cc:738] T f55615149da94dd1be6f5f09c4a2f026 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c28ff2fe9424379819d478fdd093cfc, State: Initialized, Role: FOLLOWER
I20250627 13:52:27.322438 1714 raft_consensus.cc:383] T 383a7d030ab5416abcc36eab73600269 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:27.322577 1714 raft_consensus.cc:738] T 383a7d030ab5416abcc36eab73600269 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f767210a91f44ae8b83b994131eee98, State: Initialized, Role: FOLLOWER
I20250627 13:52:27.322705 1717 consensus_queue.cc:260] T f55615149da94dd1be6f5f09c4a2f026 P 5c28ff2fe9424379819d478fdd093cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:27.323343 1717 ts_tablet_manager.cc:1428] T f55615149da94dd1be6f5f09c4a2f026 P 5c28ff2fe9424379819d478fdd093cfc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:52:27.322911 1714 consensus_queue.cc:260] T 383a7d030ab5416abcc36eab73600269 P 9f767210a91f44ae8b83b994131eee98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:27.324864 1713 tablet_bootstrap.cc:492] T 383a7d030ab5416abcc36eab73600269 P fa66786862da49ae826fb519c7d89b5e: No bootstrap required, opened a new log
I20250627 13:52:27.325034 1713 ts_tablet_manager.cc:1397] T 383a7d030ab5416abcc36eab73600269 P fa66786862da49ae826fb519c7d89b5e: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.011s
I20250627 13:52:27.325680 1714 ts_tablet_manager.cc:1428] T 383a7d030ab5416abcc36eab73600269 P 9f767210a91f44ae8b83b994131eee98: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250627 13:52:27.325994 1713 raft_consensus.cc:357] T 383a7d030ab5416abcc36eab73600269 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:27.326238 1713 raft_consensus.cc:383] T 383a7d030ab5416abcc36eab73600269 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:27.326319 1713 raft_consensus.cc:738] T 383a7d030ab5416abcc36eab73600269 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa66786862da49ae826fb519c7d89b5e, State: Initialized, Role: FOLLOWER
I20250627 13:52:27.326632 1713 consensus_queue.cc:260] T 383a7d030ab5416abcc36eab73600269 P fa66786862da49ae826fb519c7d89b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:27.328224 1713 ts_tablet_manager.cc:1428] T 383a7d030ab5416abcc36eab73600269 P fa66786862da49ae826fb519c7d89b5e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250627 13:52:27.330341 1716 tablet_bootstrap.cc:492] T 383a7d030ab5416abcc36eab73600269 P 957a8a8be3f542f48e8824e581fd0f89: No bootstrap required, opened a new log
I20250627 13:52:27.330516 1716 ts_tablet_manager.cc:1397] T 383a7d030ab5416abcc36eab73600269 P 957a8a8be3f542f48e8824e581fd0f89: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.002s
I20250627 13:52:27.331372 1716 raft_consensus.cc:357] T 383a7d030ab5416abcc36eab73600269 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:27.332399 1716 raft_consensus.cc:383] T 383a7d030ab5416abcc36eab73600269 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:27.332453 1716 raft_consensus.cc:738] T 383a7d030ab5416abcc36eab73600269 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957a8a8be3f542f48e8824e581fd0f89, State: Initialized, Role: FOLLOWER
I20250627 13:52:27.332669 1716 consensus_queue.cc:260] T 383a7d030ab5416abcc36eab73600269 P 957a8a8be3f542f48e8824e581fd0f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:27.333250 1716 ts_tablet_manager.cc:1428] T 383a7d030ab5416abcc36eab73600269 P 957a8a8be3f542f48e8824e581fd0f89: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20250627 13:52:27.335721 1597 raft_consensus.cc:491] T a0d7bf2e65de471b9867ceb7430a20d3 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:27.335870 1597 raft_consensus.cc:513] T a0d7bf2e65de471b9867ceb7430a20d3 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:27.336539 1597 leader_election.cc:290] T a0d7bf2e65de471b9867ceb7430a20d3 P fa66786862da49ae826fb519c7d89b5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023), 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69:42325)
I20250627 13:52:27.337184 598 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0d7bf2e65de471b9867ceb7430a20d3" candidate_uuid: "fa66786862da49ae826fb519c7d89b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98" is_pre_election: true
I20250627 13:52:27.337247 1123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0d7bf2e65de471b9867ceb7430a20d3" candidate_uuid: "fa66786862da49ae826fb519c7d89b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957a8a8be3f542f48e8824e581fd0f89" is_pre_election: true
I20250627 13:52:27.337383 598 raft_consensus.cc:2466] T a0d7bf2e65de471b9867ceb7430a20d3 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa66786862da49ae826fb519c7d89b5e in term 0.
I20250627 13:52:27.337426 1123 raft_consensus.cc:2466] T a0d7bf2e65de471b9867ceb7430a20d3 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa66786862da49ae826fb519c7d89b5e in term 0.
I20250627 13:52:27.337834 794 leader_election.cc:304] T a0d7bf2e65de471b9867ceb7430a20d3 P fa66786862da49ae826fb519c7d89b5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f767210a91f44ae8b83b994131eee98, fa66786862da49ae826fb519c7d89b5e; no voters:
I20250627 13:52:27.338125 1597 raft_consensus.cc:2802] T a0d7bf2e65de471b9867ceb7430a20d3 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:27.338241 1597 raft_consensus.cc:491] T a0d7bf2e65de471b9867ceb7430a20d3 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:27.338335 1597 raft_consensus.cc:3058] T a0d7bf2e65de471b9867ceb7430a20d3 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:27.340380 1597 raft_consensus.cc:513] T a0d7bf2e65de471b9867ceb7430a20d3 P fa66786862da49ae826fb519c7d89b5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:27.340826 1597 leader_election.cc:290] T a0d7bf2e65de471b9867ceb7430a20d3 P fa66786862da49ae826fb519c7d89b5e [CANDIDATE]: Term 1 election: Requested vote from peers 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023), 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69:42325)
I20250627 13:52:27.341356 1123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0d7bf2e65de471b9867ceb7430a20d3" candidate_uuid: "fa66786862da49ae826fb519c7d89b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957a8a8be3f542f48e8824e581fd0f89"
I20250627 13:52:27.341554 1123 raft_consensus.cc:3058] T a0d7bf2e65de471b9867ceb7430a20d3 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:27.343587 1123 raft_consensus.cc:2466] T a0d7bf2e65de471b9867ceb7430a20d3 P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa66786862da49ae826fb519c7d89b5e in term 1.
I20250627 13:52:27.343961 598 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a0d7bf2e65de471b9867ceb7430a20d3" candidate_uuid: "fa66786862da49ae826fb519c7d89b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98"
I20250627 13:52:27.344148 598 raft_consensus.cc:3058] T a0d7bf2e65de471b9867ceb7430a20d3 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:27.344126 797 leader_election.cc:304] T a0d7bf2e65de471b9867ceb7430a20d3 P fa66786862da49ae826fb519c7d89b5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, fa66786862da49ae826fb519c7d89b5e; no voters:
I20250627 13:52:27.344427 1597 raft_consensus.cc:2802] T a0d7bf2e65de471b9867ceb7430a20d3 P fa66786862da49ae826fb519c7d89b5e [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:27.344905 1597 raft_consensus.cc:695] T a0d7bf2e65de471b9867ceb7430a20d3 P fa66786862da49ae826fb519c7d89b5e [term 1 LEADER]: Becoming Leader. State: Replica: fa66786862da49ae826fb519c7d89b5e, State: Running, Role: LEADER
I20250627 13:52:27.345237 1597 consensus_queue.cc:237] T a0d7bf2e65de471b9867ceb7430a20d3 P fa66786862da49ae826fb519c7d89b5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:27.346824 598 raft_consensus.cc:2466] T a0d7bf2e65de471b9867ceb7430a20d3 P 9f767210a91f44ae8b83b994131eee98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa66786862da49ae826fb519c7d89b5e in term 1.
I20250627 13:52:27.348268 453 catalog_manager.cc:5582] T a0d7bf2e65de471b9867ceb7430a20d3 P fa66786862da49ae826fb519c7d89b5e reported cstate change: term changed from 0 to 1, leader changed from <none> to fa66786862da49ae826fb519c7d89b5e (127.0.103.67). New cstate: current_term: 1 leader_uuid: "fa66786862da49ae826fb519c7d89b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } health_report { overall_health: UNKNOWN } } }
I20250627 13:52:27.409358 1689 raft_consensus.cc:491] T f55615149da94dd1be6f5f09c4a2f026 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:27.409528 1689 raft_consensus.cc:513] T f55615149da94dd1be6f5f09c4a2f026 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:27.410148 1689 leader_election.cc:290] T f55615149da94dd1be6f5f09c4a2f026 P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499), 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70:39887)
I20250627 13:52:27.410612 730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55615149da94dd1be6f5f09c4a2f026" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c4f59060974a3b823305b9dd339a7e" is_pre_election: true
I20250627 13:52:27.410707 1253 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55615149da94dd1be6f5f09c4a2f026" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c28ff2fe9424379819d478fdd093cfc" is_pre_election: true
I20250627 13:52:27.410828 730 raft_consensus.cc:2466] T f55615149da94dd1be6f5f09c4a2f026 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 0.
I20250627 13:52:27.410921 1253 raft_consensus.cc:2466] T f55615149da94dd1be6f5f09c4a2f026 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 0.
I20250627 13:52:27.411289 928 leader_election.cc:304] T f55615149da94dd1be6f5f09c4a2f026 P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50e8c97891d74b38b5c130d3bcbabcb6, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:27.411589 1689 raft_consensus.cc:2802] T f55615149da94dd1be6f5f09c4a2f026 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:27.411708 1689 raft_consensus.cc:491] T f55615149da94dd1be6f5f09c4a2f026 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:27.411779 1689 raft_consensus.cc:3058] T f55615149da94dd1be6f5f09c4a2f026 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:27.413630 1689 raft_consensus.cc:513] T f55615149da94dd1be6f5f09c4a2f026 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:27.414206 1689 leader_election.cc:290] T f55615149da94dd1be6f5f09c4a2f026 P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 election: Requested vote from peers 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499), 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70:39887)
I20250627 13:52:27.414664 730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55615149da94dd1be6f5f09c4a2f026" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c4f59060974a3b823305b9dd339a7e"
I20250627 13:52:27.414680 1253 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55615149da94dd1be6f5f09c4a2f026" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c28ff2fe9424379819d478fdd093cfc"
I20250627 13:52:27.414863 1253 raft_consensus.cc:3058] T f55615149da94dd1be6f5f09c4a2f026 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:27.414881 730 raft_consensus.cc:3058] T f55615149da94dd1be6f5f09c4a2f026 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:27.417797 730 raft_consensus.cc:2466] T f55615149da94dd1be6f5f09c4a2f026 P 75c4f59060974a3b823305b9dd339a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 1.
I20250627 13:52:27.417807 1253 raft_consensus.cc:2466] T f55615149da94dd1be6f5f09c4a2f026 P 5c28ff2fe9424379819d478fdd093cfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 1.
I20250627 13:52:27.418255 928 leader_election.cc:304] T f55615149da94dd1be6f5f09c4a2f026 P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 50e8c97891d74b38b5c130d3bcbabcb6, 5c28ff2fe9424379819d478fdd093cfc, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:27.418524 1689 raft_consensus.cc:2802] T f55615149da94dd1be6f5f09c4a2f026 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:27.418870 1689 raft_consensus.cc:695] T f55615149da94dd1be6f5f09c4a2f026 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 LEADER]: Becoming Leader. State: Replica: 50e8c97891d74b38b5c130d3bcbabcb6, State: Running, Role: LEADER
I20250627 13:52:27.419116 1689 consensus_queue.cc:237] T f55615149da94dd1be6f5f09c4a2f026 P 50e8c97891d74b38b5c130d3bcbabcb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:27.421619 453 catalog_manager.cc:5582] T f55615149da94dd1be6f5f09c4a2f026 P 50e8c97891d74b38b5c130d3bcbabcb6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68). New cstate: current_term: 1 leader_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } health_report { overall_health: HEALTHY } } }
I20250627 13:52:27.432111 1597 raft_consensus.cc:491] T 383a7d030ab5416abcc36eab73600269 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:27.432376 1597 raft_consensus.cc:513] T 383a7d030ab5416abcc36eab73600269 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:27.433132 1597 leader_election.cc:290] T 383a7d030ab5416abcc36eab73600269 P fa66786862da49ae826fb519c7d89b5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023), 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69:42325)
I20250627 13:52:27.433781 1123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "383a7d030ab5416abcc36eab73600269" candidate_uuid: "fa66786862da49ae826fb519c7d89b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957a8a8be3f542f48e8824e581fd0f89" is_pre_election: true
I20250627 13:52:27.433812 598 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "383a7d030ab5416abcc36eab73600269" candidate_uuid: "fa66786862da49ae826fb519c7d89b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98" is_pre_election: true
I20250627 13:52:27.433990 1123 raft_consensus.cc:2466] T 383a7d030ab5416abcc36eab73600269 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa66786862da49ae826fb519c7d89b5e in term 0.
I20250627 13:52:27.434077 598 raft_consensus.cc:2466] T 383a7d030ab5416abcc36eab73600269 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa66786862da49ae826fb519c7d89b5e in term 0.
I20250627 13:52:27.434487 797 leader_election.cc:304] T 383a7d030ab5416abcc36eab73600269 P fa66786862da49ae826fb519c7d89b5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, 9f767210a91f44ae8b83b994131eee98, fa66786862da49ae826fb519c7d89b5e; no voters:
I20250627 13:52:27.434759 1597 raft_consensus.cc:2802] T 383a7d030ab5416abcc36eab73600269 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:27.434849 1597 raft_consensus.cc:491] T 383a7d030ab5416abcc36eab73600269 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:27.434945 1597 raft_consensus.cc:3058] T 383a7d030ab5416abcc36eab73600269 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:27.436748 1597 raft_consensus.cc:513] T 383a7d030ab5416abcc36eab73600269 P fa66786862da49ae826fb519c7d89b5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:27.437251 1597 leader_election.cc:290] T 383a7d030ab5416abcc36eab73600269 P fa66786862da49ae826fb519c7d89b5e [CANDIDATE]: Term 1 election: Requested vote from peers 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023), 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69:42325)
I20250627 13:52:27.437620 598 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "383a7d030ab5416abcc36eab73600269" candidate_uuid: "fa66786862da49ae826fb519c7d89b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98"
I20250627 13:52:27.437747 1123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "383a7d030ab5416abcc36eab73600269" candidate_uuid: "fa66786862da49ae826fb519c7d89b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957a8a8be3f542f48e8824e581fd0f89"
I20250627 13:52:27.437824 598 raft_consensus.cc:3058] T 383a7d030ab5416abcc36eab73600269 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:27.437935 1123 raft_consensus.cc:3058] T 383a7d030ab5416abcc36eab73600269 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:27.439749 598 raft_consensus.cc:2466] T 383a7d030ab5416abcc36eab73600269 P 9f767210a91f44ae8b83b994131eee98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa66786862da49ae826fb519c7d89b5e in term 1.
I20250627 13:52:27.439899 1123 raft_consensus.cc:2466] T 383a7d030ab5416abcc36eab73600269 P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa66786862da49ae826fb519c7d89b5e in term 1.
I20250627 13:52:27.440124 794 leader_election.cc:304] T 383a7d030ab5416abcc36eab73600269 P fa66786862da49ae826fb519c7d89b5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f767210a91f44ae8b83b994131eee98, fa66786862da49ae826fb519c7d89b5e; no voters:
I20250627 13:52:27.440439 1597 raft_consensus.cc:2802] T 383a7d030ab5416abcc36eab73600269 P fa66786862da49ae826fb519c7d89b5e [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:27.440610 1597 raft_consensus.cc:695] T 383a7d030ab5416abcc36eab73600269 P fa66786862da49ae826fb519c7d89b5e [term 1 LEADER]: Becoming Leader. State: Replica: fa66786862da49ae826fb519c7d89b5e, State: Running, Role: LEADER
I20250627 13:52:27.440848 1597 consensus_queue.cc:237] T 383a7d030ab5416abcc36eab73600269 P fa66786862da49ae826fb519c7d89b5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:27.443476 453 catalog_manager.cc:5582] T 383a7d030ab5416abcc36eab73600269 P fa66786862da49ae826fb519c7d89b5e reported cstate change: term changed from 0 to 1, leader changed from <none> to fa66786862da49ae826fb519c7d89b5e (127.0.103.67). New cstate: current_term: 1 leader_uuid: "fa66786862da49ae826fb519c7d89b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } health_report { overall_health: UNKNOWN } } }
I20250627 13:52:27.775684 1597 consensus_queue.cc:1035] T a0d7bf2e65de471b9867ceb7430a20d3 P fa66786862da49ae826fb519c7d89b5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:27.781663 1597 consensus_queue.cc:1035] T a0d7bf2e65de471b9867ceb7430a20d3 P fa66786862da49ae826fb519c7d89b5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250627 13:52:27.785583 1735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:27.786024 1735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:27.794219 1735 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20250627 13:52:27.879354 1689 consensus_queue.cc:1035] T f55615149da94dd1be6f5f09c4a2f026 P 50e8c97891d74b38b5c130d3bcbabcb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:27.885432 1689 consensus_queue.cc:1035] T f55615149da94dd1be6f5f09c4a2f026 P 50e8c97891d74b38b5c130d3bcbabcb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:27.917116 1597 consensus_queue.cc:1035] T 383a7d030ab5416abcc36eab73600269 P fa66786862da49ae826fb519c7d89b5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:27.922593 1730 consensus_queue.cc:1035] T 383a7d030ab5416abcc36eab73600269 P fa66786862da49ae826fb519c7d89b5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 235.528 ms
time per row: 0.117764 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20250627 13:52:29.178050 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49852:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250627 13:52:29.186605 972 tablet_service.cc:1468] Processing CreateTablet for tablet b77a6923c0494a5ea8471b899acff4df (DEFAULT_TABLE table=rebalance_test_table_5 [id=423c14db82b44ecd85b47d15ca777bff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:29.187044 972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b77a6923c0494a5ea8471b899acff4df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:29.187979 709 tablet_service.cc:1468] Processing CreateTablet for tablet b77a6923c0494a5ea8471b899acff4df (DEFAULT_TABLE table=rebalance_test_table_5 [id=423c14db82b44ecd85b47d15ca777bff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:29.188380 709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b77a6923c0494a5ea8471b899acff4df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:29.188866 840 tablet_service.cc:1468] Processing CreateTablet for tablet c007bb30602244e6b67ecb5970a07964 (DEFAULT_TABLE table=rebalance_test_table_5 [id=423c14db82b44ecd85b47d15ca777bff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:29.189297 840 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c007bb30602244e6b67ecb5970a07964. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:29.189461 710 tablet_service.cc:1468] Processing CreateTablet for tablet ccfdf884041949bfbfa6a09574369a13 (DEFAULT_TABLE table=rebalance_test_table_5 [id=423c14db82b44ecd85b47d15ca777bff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:29.190078 1102 tablet_service.cc:1468] Processing CreateTablet for tablet c007bb30602244e6b67ecb5970a07964 (DEFAULT_TABLE table=rebalance_test_table_5 [id=423c14db82b44ecd85b47d15ca777bff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:29.190461 1102 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c007bb30602244e6b67ecb5970a07964. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:29.193451 971 tablet_service.cc:1468] Processing CreateTablet for tablet ccfdf884041949bfbfa6a09574369a13 (DEFAULT_TABLE table=rebalance_test_table_5 [id=423c14db82b44ecd85b47d15ca777bff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:29.193861 971 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccfdf884041949bfbfa6a09574369a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:29.194914 1234 tablet_service.cc:1468] Processing CreateTablet for tablet b77a6923c0494a5ea8471b899acff4df (DEFAULT_TABLE table=rebalance_test_table_5 [id=423c14db82b44ecd85b47d15ca777bff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:29.195287 1234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b77a6923c0494a5ea8471b899acff4df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:29.196653 1233 tablet_service.cc:1468] Processing CreateTablet for tablet ccfdf884041949bfbfa6a09574369a13 (DEFAULT_TABLE table=rebalance_test_table_5 [id=423c14db82b44ecd85b47d15ca777bff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:29.197005 1233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccfdf884041949bfbfa6a09574369a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:29.198194 710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccfdf884041949bfbfa6a09574369a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:29.198866 1805 tablet_bootstrap.cc:492] T b77a6923c0494a5ea8471b899acff4df P 50e8c97891d74b38b5c130d3bcbabcb6: Bootstrap starting.
I20250627 13:52:29.201071 1805 tablet_bootstrap.cc:654] T b77a6923c0494a5ea8471b899acff4df P 50e8c97891d74b38b5c130d3bcbabcb6: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:29.201858 578 tablet_service.cc:1468] Processing CreateTablet for tablet c007bb30602244e6b67ecb5970a07964 (DEFAULT_TABLE table=rebalance_test_table_5 [id=423c14db82b44ecd85b47d15ca777bff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:29.202250 578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c007bb30602244e6b67ecb5970a07964. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:29.215560 1810 tablet_bootstrap.cc:492] T c007bb30602244e6b67ecb5970a07964 P 957a8a8be3f542f48e8824e581fd0f89: Bootstrap starting.
I20250627 13:52:29.220438 1810 tablet_bootstrap.cc:654] T c007bb30602244e6b67ecb5970a07964 P 957a8a8be3f542f48e8824e581fd0f89: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:29.223639 1811 tablet_bootstrap.cc:492] T b77a6923c0494a5ea8471b899acff4df P 5c28ff2fe9424379819d478fdd093cfc: Bootstrap starting.
I20250627 13:52:29.224309 1809 tablet_bootstrap.cc:492] T c007bb30602244e6b67ecb5970a07964 P 9f767210a91f44ae8b83b994131eee98: Bootstrap starting.
I20250627 13:52:29.225966 1808 tablet_bootstrap.cc:492] T ccfdf884041949bfbfa6a09574369a13 P 75c4f59060974a3b823305b9dd339a7e: Bootstrap starting.
I20250627 13:52:29.226619 1809 tablet_bootstrap.cc:654] T c007bb30602244e6b67ecb5970a07964 P 9f767210a91f44ae8b83b994131eee98: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:29.228060 1808 tablet_bootstrap.cc:654] T ccfdf884041949bfbfa6a09574369a13 P 75c4f59060974a3b823305b9dd339a7e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:29.228840 1811 tablet_bootstrap.cc:654] T b77a6923c0494a5ea8471b899acff4df P 5c28ff2fe9424379819d478fdd093cfc: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:29.228960 1805 tablet_bootstrap.cc:492] T b77a6923c0494a5ea8471b899acff4df P 50e8c97891d74b38b5c130d3bcbabcb6: No bootstrap required, opened a new log
I20250627 13:52:29.229123 1805 ts_tablet_manager.cc:1397] T b77a6923c0494a5ea8471b899acff4df P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent bootstrapping tablet: real 0.030s user 0.017s sys 0.013s
I20250627 13:52:29.229418 1806 tablet_bootstrap.cc:492] T c007bb30602244e6b67ecb5970a07964 P fa66786862da49ae826fb519c7d89b5e: Bootstrap starting.
I20250627 13:52:29.230036 1805 raft_consensus.cc:357] T b77a6923c0494a5ea8471b899acff4df P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:29.230255 1805 raft_consensus.cc:383] T b77a6923c0494a5ea8471b899acff4df P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:29.230307 1809 tablet_bootstrap.cc:492] T c007bb30602244e6b67ecb5970a07964 P 9f767210a91f44ae8b83b994131eee98: No bootstrap required, opened a new log
I20250627 13:52:29.230386 1805 raft_consensus.cc:738] T b77a6923c0494a5ea8471b899acff4df P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e8c97891d74b38b5c130d3bcbabcb6, State: Initialized, Role: FOLLOWER
I20250627 13:52:29.230618 1805 consensus_queue.cc:260] T b77a6923c0494a5ea8471b899acff4df P 50e8c97891d74b38b5c130d3bcbabcb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:29.230887 1810 tablet_bootstrap.cc:492] T c007bb30602244e6b67ecb5970a07964 P 957a8a8be3f542f48e8824e581fd0f89: No bootstrap required, opened a new log
I20250627 13:52:29.231050 1810 ts_tablet_manager.cc:1397] T c007bb30602244e6b67ecb5970a07964 P 957a8a8be3f542f48e8824e581fd0f89: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20250627 13:52:29.231222 1809 ts_tablet_manager.cc:1397] T c007bb30602244e6b67ecb5970a07964 P 9f767210a91f44ae8b83b994131eee98: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20250627 13:52:29.231841 1805 ts_tablet_manager.cc:1428] T b77a6923c0494a5ea8471b899acff4df P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250627 13:52:29.232117 1805 tablet_bootstrap.cc:492] T ccfdf884041949bfbfa6a09574369a13 P 50e8c97891d74b38b5c130d3bcbabcb6: Bootstrap starting.
I20250627 13:52:29.232728 1809 raft_consensus.cc:357] T c007bb30602244e6b67ecb5970a07964 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:29.233043 1810 raft_consensus.cc:357] T c007bb30602244e6b67ecb5970a07964 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:29.233345 1810 raft_consensus.cc:383] T c007bb30602244e6b67ecb5970a07964 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:29.233489 1809 raft_consensus.cc:383] T c007bb30602244e6b67ecb5970a07964 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:29.233742 1809 raft_consensus.cc:738] T c007bb30602244e6b67ecb5970a07964 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f767210a91f44ae8b83b994131eee98, State: Initialized, Role: FOLLOWER
I20250627 13:52:29.234185 1806 tablet_bootstrap.cc:654] T c007bb30602244e6b67ecb5970a07964 P fa66786862da49ae826fb519c7d89b5e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:29.234915 1810 raft_consensus.cc:738] T c007bb30602244e6b67ecb5970a07964 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957a8a8be3f542f48e8824e581fd0f89, State: Initialized, Role: FOLLOWER
I20250627 13:52:29.235473 1810 consensus_queue.cc:260] T c007bb30602244e6b67ecb5970a07964 P 957a8a8be3f542f48e8824e581fd0f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:29.235630 1805 tablet_bootstrap.cc:654] T ccfdf884041949bfbfa6a09574369a13 P 50e8c97891d74b38b5c130d3bcbabcb6: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:29.235823 1809 consensus_queue.cc:260] T c007bb30602244e6b67ecb5970a07964 P 9f767210a91f44ae8b83b994131eee98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:29.236486 1808 tablet_bootstrap.cc:492] T ccfdf884041949bfbfa6a09574369a13 P 75c4f59060974a3b823305b9dd339a7e: No bootstrap required, opened a new log
I20250627 13:52:29.236537 1810 ts_tablet_manager.cc:1428] T c007bb30602244e6b67ecb5970a07964 P 957a8a8be3f542f48e8824e581fd0f89: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250627 13:52:29.236642 1808 ts_tablet_manager.cc:1397] T ccfdf884041949bfbfa6a09574369a13 P 75c4f59060974a3b823305b9dd339a7e: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.002s
I20250627 13:52:29.237552 1808 raft_consensus.cc:357] T ccfdf884041949bfbfa6a09574369a13 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:29.237869 1808 raft_consensus.cc:383] T ccfdf884041949bfbfa6a09574369a13 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:29.238018 1808 raft_consensus.cc:738] T ccfdf884041949bfbfa6a09574369a13 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c4f59060974a3b823305b9dd339a7e, State: Initialized, Role: FOLLOWER
I20250627 13:52:29.238359 1808 consensus_queue.cc:260] T ccfdf884041949bfbfa6a09574369a13 P 75c4f59060974a3b823305b9dd339a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:29.238834 1809 ts_tablet_manager.cc:1428] T c007bb30602244e6b67ecb5970a07964 P 9f767210a91f44ae8b83b994131eee98: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20250627 13:52:29.239715 1808 ts_tablet_manager.cc:1428] T ccfdf884041949bfbfa6a09574369a13 P 75c4f59060974a3b823305b9dd339a7e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250627 13:52:29.240032 1808 tablet_bootstrap.cc:492] T b77a6923c0494a5ea8471b899acff4df P 75c4f59060974a3b823305b9dd339a7e: Bootstrap starting.
I20250627 13:52:29.241099 1811 tablet_bootstrap.cc:492] T b77a6923c0494a5ea8471b899acff4df P 5c28ff2fe9424379819d478fdd093cfc: No bootstrap required, opened a new log
I20250627 13:52:29.241293 1811 ts_tablet_manager.cc:1397] T b77a6923c0494a5ea8471b899acff4df P 5c28ff2fe9424379819d478fdd093cfc: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20250627 13:52:29.241794 1808 tablet_bootstrap.cc:654] T b77a6923c0494a5ea8471b899acff4df P 75c4f59060974a3b823305b9dd339a7e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:29.243160 1811 raft_consensus.cc:357] T b77a6923c0494a5ea8471b899acff4df P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:29.243412 1811 raft_consensus.cc:383] T b77a6923c0494a5ea8471b899acff4df P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:29.243518 1811 raft_consensus.cc:738] T b77a6923c0494a5ea8471b899acff4df P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c28ff2fe9424379819d478fdd093cfc, State: Initialized, Role: FOLLOWER
I20250627 13:52:29.243780 1811 consensus_queue.cc:260] T b77a6923c0494a5ea8471b899acff4df P 5c28ff2fe9424379819d478fdd093cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:29.245843 1811 ts_tablet_manager.cc:1428] T b77a6923c0494a5ea8471b899acff4df P 5c28ff2fe9424379819d478fdd093cfc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250627 13:52:29.246229 1811 tablet_bootstrap.cc:492] T ccfdf884041949bfbfa6a09574369a13 P 5c28ff2fe9424379819d478fdd093cfc: Bootstrap starting.
I20250627 13:52:29.248462 1811 tablet_bootstrap.cc:654] T ccfdf884041949bfbfa6a09574369a13 P 5c28ff2fe9424379819d478fdd093cfc: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:29.255532 1816 raft_consensus.cc:491] T c007bb30602244e6b67ecb5970a07964 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:29.255739 1816 raft_consensus.cc:513] T c007bb30602244e6b67ecb5970a07964 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:29.256649 1816 leader_election.cc:290] T c007bb30602244e6b67ecb5970a07964 P 9f767210a91f44ae8b83b994131eee98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747), 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69:42325)
I20250627 13:52:29.260818 1122 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c007bb30602244e6b67ecb5970a07964" candidate_uuid: "9f767210a91f44ae8b83b994131eee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957a8a8be3f542f48e8824e581fd0f89" is_pre_election: true
I20250627 13:52:29.261054 1122 raft_consensus.cc:2466] T c007bb30602244e6b67ecb5970a07964 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f767210a91f44ae8b83b994131eee98 in term 0.
I20250627 13:52:29.261260 861 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c007bb30602244e6b67ecb5970a07964" candidate_uuid: "9f767210a91f44ae8b83b994131eee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e" is_pre_election: true
I20250627 13:52:29.261462 1805 tablet_bootstrap.cc:492] T ccfdf884041949bfbfa6a09574369a13 P 50e8c97891d74b38b5c130d3bcbabcb6: No bootstrap required, opened a new log
I20250627 13:52:29.261708 1805 ts_tablet_manager.cc:1397] T ccfdf884041949bfbfa6a09574369a13 P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.017s
W20250627 13:52:29.262228 533 leader_election.cc:343] T c007bb30602244e6b67ecb5970a07964 P 9f767210a91f44ae8b83b994131eee98 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:52:29.262606 534 leader_election.cc:304] T c007bb30602244e6b67ecb5970a07964 P 9f767210a91f44ae8b83b994131eee98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, 9f767210a91f44ae8b83b994131eee98; no voters: fa66786862da49ae826fb519c7d89b5e
I20250627 13:52:29.263053 1816 raft_consensus.cc:2802] T c007bb30602244e6b67ecb5970a07964 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:29.263183 1816 raft_consensus.cc:491] T c007bb30602244e6b67ecb5970a07964 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:29.263280 1816 raft_consensus.cc:3058] T c007bb30602244e6b67ecb5970a07964 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:29.264863 1805 raft_consensus.cc:357] T ccfdf884041949bfbfa6a09574369a13 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:29.265116 1805 raft_consensus.cc:383] T ccfdf884041949bfbfa6a09574369a13 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:29.265244 1805 raft_consensus.cc:738] T ccfdf884041949bfbfa6a09574369a13 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e8c97891d74b38b5c130d3bcbabcb6, State: Initialized, Role: FOLLOWER
I20250627 13:52:29.265534 1805 consensus_queue.cc:260] T ccfdf884041949bfbfa6a09574369a13 P 50e8c97891d74b38b5c130d3bcbabcb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:29.266252 1805 ts_tablet_manager.cc:1428] T ccfdf884041949bfbfa6a09574369a13 P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250627 13:52:29.271999 1816 raft_consensus.cc:513] T c007bb30602244e6b67ecb5970a07964 P 9f767210a91f44ae8b83b994131eee98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:29.273011 1811 tablet_bootstrap.cc:492] T ccfdf884041949bfbfa6a09574369a13 P 5c28ff2fe9424379819d478fdd093cfc: No bootstrap required, opened a new log
I20250627 13:52:29.273320 1811 ts_tablet_manager.cc:1397] T ccfdf884041949bfbfa6a09574369a13 P 5c28ff2fe9424379819d478fdd093cfc: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.013s
I20250627 13:52:29.274000 1122 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c007bb30602244e6b67ecb5970a07964" candidate_uuid: "9f767210a91f44ae8b83b994131eee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957a8a8be3f542f48e8824e581fd0f89"
I20250627 13:52:29.274224 1122 raft_consensus.cc:3058] T c007bb30602244e6b67ecb5970a07964 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:29.274188 861 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c007bb30602244e6b67ecb5970a07964" candidate_uuid: "9f767210a91f44ae8b83b994131eee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e"
I20250627 13:52:29.274411 1811 raft_consensus.cc:357] T ccfdf884041949bfbfa6a09574369a13 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:29.274642 1811 raft_consensus.cc:383] T ccfdf884041949bfbfa6a09574369a13 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:29.274718 1811 raft_consensus.cc:738] T ccfdf884041949bfbfa6a09574369a13 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c28ff2fe9424379819d478fdd093cfc, State: Initialized, Role: FOLLOWER
W20250627 13:52:29.274722 533 leader_election.cc:343] T c007bb30602244e6b67ecb5970a07964 P 9f767210a91f44ae8b83b994131eee98 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:52:29.274977 1811 consensus_queue.cc:260] T ccfdf884041949bfbfa6a09574369a13 P 5c28ff2fe9424379819d478fdd093cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:29.275621 1811 ts_tablet_manager.cc:1428] T ccfdf884041949bfbfa6a09574369a13 P 5c28ff2fe9424379819d478fdd093cfc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250627 13:52:29.275956 1816 leader_election.cc:290] T c007bb30602244e6b67ecb5970a07964 P 9f767210a91f44ae8b83b994131eee98 [CANDIDATE]: Term 1 election: Requested vote from peers fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747), 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69:42325)
I20250627 13:52:29.277065 1122 raft_consensus.cc:2466] T c007bb30602244e6b67ecb5970a07964 P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f767210a91f44ae8b83b994131eee98 in term 1.
I20250627 13:52:29.277459 534 leader_election.cc:304] T c007bb30602244e6b67ecb5970a07964 P 9f767210a91f44ae8b83b994131eee98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, 9f767210a91f44ae8b83b994131eee98; no voters: fa66786862da49ae826fb519c7d89b5e
I20250627 13:52:29.277940 1816 raft_consensus.cc:2802] T c007bb30602244e6b67ecb5970a07964 P 9f767210a91f44ae8b83b994131eee98 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:29.278169 1806 tablet_bootstrap.cc:492] T c007bb30602244e6b67ecb5970a07964 P fa66786862da49ae826fb519c7d89b5e: No bootstrap required, opened a new log
I20250627 13:52:29.278316 1806 ts_tablet_manager.cc:1397] T c007bb30602244e6b67ecb5970a07964 P fa66786862da49ae826fb519c7d89b5e: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.012s
I20250627 13:52:29.278424 1808 tablet_bootstrap.cc:492] T b77a6923c0494a5ea8471b899acff4df P 75c4f59060974a3b823305b9dd339a7e: No bootstrap required, opened a new log
I20250627 13:52:29.278599 1808 ts_tablet_manager.cc:1397] T b77a6923c0494a5ea8471b899acff4df P 75c4f59060974a3b823305b9dd339a7e: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.015s
I20250627 13:52:29.279101 1806 raft_consensus.cc:357] T c007bb30602244e6b67ecb5970a07964 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:29.279341 1806 raft_consensus.cc:383] T c007bb30602244e6b67ecb5970a07964 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:29.279317 1808 raft_consensus.cc:357] T b77a6923c0494a5ea8471b899acff4df P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:29.279449 1806 raft_consensus.cc:738] T c007bb30602244e6b67ecb5970a07964 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa66786862da49ae826fb519c7d89b5e, State: Initialized, Role: FOLLOWER
I20250627 13:52:29.279662 1806 consensus_queue.cc:260] T c007bb30602244e6b67ecb5970a07964 P fa66786862da49ae826fb519c7d89b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:29.280187 1806 ts_tablet_manager.cc:1428] T c007bb30602244e6b67ecb5970a07964 P fa66786862da49ae826fb519c7d89b5e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250627 13:52:29.280436 1808 raft_consensus.cc:383] T b77a6923c0494a5ea8471b899acff4df P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:29.280560 1808 raft_consensus.cc:738] T b77a6923c0494a5ea8471b899acff4df P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c4f59060974a3b823305b9dd339a7e, State: Initialized, Role: FOLLOWER
I20250627 13:52:29.280910 1808 consensus_queue.cc:260] T b77a6923c0494a5ea8471b899acff4df P 75c4f59060974a3b823305b9dd339a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:29.281385 1808 ts_tablet_manager.cc:1428] T b77a6923c0494a5ea8471b899acff4df P 75c4f59060974a3b823305b9dd339a7e: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250627 13:52:29.281808 1816 raft_consensus.cc:695] T c007bb30602244e6b67ecb5970a07964 P 9f767210a91f44ae8b83b994131eee98 [term 1 LEADER]: Becoming Leader. State: Replica: 9f767210a91f44ae8b83b994131eee98, State: Running, Role: LEADER
I20250627 13:52:29.282334 1816 consensus_queue.cc:237] T c007bb30602244e6b67ecb5970a07964 P 9f767210a91f44ae8b83b994131eee98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:29.288578 454 catalog_manager.cc:5582] T c007bb30602244e6b67ecb5970a07964 P 9f767210a91f44ae8b83b994131eee98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f767210a91f44ae8b83b994131eee98 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "9f767210a91f44ae8b83b994131eee98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } health_report { overall_health: HEALTHY } } }
I20250627 13:52:29.362671 1732 raft_consensus.cc:491] T b77a6923c0494a5ea8471b899acff4df P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:29.362846 1732 raft_consensus.cc:513] T b77a6923c0494a5ea8471b899acff4df P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:29.363401 1732 leader_election.cc:290] T b77a6923c0494a5ea8471b899acff4df P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499), 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70:39887)
I20250627 13:52:29.363889 1253 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b77a6923c0494a5ea8471b899acff4df" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c28ff2fe9424379819d478fdd093cfc" is_pre_election: true
I20250627 13:52:29.363889 730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b77a6923c0494a5ea8471b899acff4df" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c4f59060974a3b823305b9dd339a7e" is_pre_election: true
I20250627 13:52:29.364097 730 raft_consensus.cc:2466] T b77a6923c0494a5ea8471b899acff4df P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 0.
I20250627 13:52:29.364118 1253 raft_consensus.cc:2466] T b77a6923c0494a5ea8471b899acff4df P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 0.
I20250627 13:52:29.364581 928 leader_election.cc:304] T b77a6923c0494a5ea8471b899acff4df P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 50e8c97891d74b38b5c130d3bcbabcb6, 5c28ff2fe9424379819d478fdd093cfc, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:29.364871 1732 raft_consensus.cc:2802] T b77a6923c0494a5ea8471b899acff4df P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:29.365000 1732 raft_consensus.cc:491] T b77a6923c0494a5ea8471b899acff4df P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:29.365088 1732 raft_consensus.cc:3058] T b77a6923c0494a5ea8471b899acff4df P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:29.367025 1732 raft_consensus.cc:513] T b77a6923c0494a5ea8471b899acff4df P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:29.367476 1732 leader_election.cc:290] T b77a6923c0494a5ea8471b899acff4df P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 election: Requested vote from peers 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499), 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70:39887)
I20250627 13:52:29.367857 730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b77a6923c0494a5ea8471b899acff4df" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c4f59060974a3b823305b9dd339a7e"
I20250627 13:52:29.367873 1253 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b77a6923c0494a5ea8471b899acff4df" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c28ff2fe9424379819d478fdd093cfc"
I20250627 13:52:29.368026 730 raft_consensus.cc:3058] T b77a6923c0494a5ea8471b899acff4df P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:29.368027 1253 raft_consensus.cc:3058] T b77a6923c0494a5ea8471b899acff4df P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:29.369886 730 raft_consensus.cc:2466] T b77a6923c0494a5ea8471b899acff4df P 75c4f59060974a3b823305b9dd339a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 1.
I20250627 13:52:29.369918 1253 raft_consensus.cc:2466] T b77a6923c0494a5ea8471b899acff4df P 5c28ff2fe9424379819d478fdd093cfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 1.
I20250627 13:52:29.370261 928 leader_election.cc:304] T b77a6923c0494a5ea8471b899acff4df P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 50e8c97891d74b38b5c130d3bcbabcb6, 5c28ff2fe9424379819d478fdd093cfc, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:29.370536 1732 raft_consensus.cc:2802] T b77a6923c0494a5ea8471b899acff4df P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:29.370858 1732 raft_consensus.cc:695] T b77a6923c0494a5ea8471b899acff4df P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 LEADER]: Becoming Leader. State: Replica: 50e8c97891d74b38b5c130d3bcbabcb6, State: Running, Role: LEADER
I20250627 13:52:29.371141 1732 consensus_queue.cc:237] T b77a6923c0494a5ea8471b899acff4df P 50e8c97891d74b38b5c130d3bcbabcb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:29.373729 454 catalog_manager.cc:5582] T b77a6923c0494a5ea8471b899acff4df P 50e8c97891d74b38b5c130d3bcbabcb6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68). New cstate: current_term: 1 leader_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } health_report { overall_health: UNKNOWN } } }
I20250627 13:52:29.380201 1790 raft_consensus.cc:491] T c007bb30602244e6b67ecb5970a07964 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:29.380352 1790 raft_consensus.cc:513] T c007bb30602244e6b67ecb5970a07964 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:29.380915 1790 leader_election.cc:290] T c007bb30602244e6b67ecb5970a07964 P fa66786862da49ae826fb519c7d89b5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69:42325), 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023)
I20250627 13:52:29.381351 1122 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c007bb30602244e6b67ecb5970a07964" candidate_uuid: "fa66786862da49ae826fb519c7d89b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957a8a8be3f542f48e8824e581fd0f89" is_pre_election: true
I20250627 13:52:29.381363 597 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c007bb30602244e6b67ecb5970a07964" candidate_uuid: "fa66786862da49ae826fb519c7d89b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98" is_pre_election: true
I20250627 13:52:29.381594 1122 raft_consensus.cc:2391] T c007bb30602244e6b67ecb5970a07964 P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fa66786862da49ae826fb519c7d89b5e in current term 1: Already voted for candidate 9f767210a91f44ae8b83b994131eee98 in this term.
I20250627 13:52:29.381980 797 leader_election.cc:304] T c007bb30602244e6b67ecb5970a07964 P fa66786862da49ae826fb519c7d89b5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa66786862da49ae826fb519c7d89b5e; no voters: 957a8a8be3f542f48e8824e581fd0f89, 9f767210a91f44ae8b83b994131eee98
I20250627 13:52:29.382217 1790 raft_consensus.cc:3058] T c007bb30602244e6b67ecb5970a07964 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:29.383953 1790 raft_consensus.cc:2747] T c007bb30602244e6b67ecb5970a07964 P fa66786862da49ae826fb519c7d89b5e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250627 13:52:29.590971 1342 raft_consensus.cc:491] T ccfdf884041949bfbfa6a09574369a13 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:29.591225 1342 raft_consensus.cc:513] T ccfdf884041949bfbfa6a09574369a13 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:29.591862 1342 leader_election.cc:290] T ccfdf884041949bfbfa6a09574369a13 P 5c28ff2fe9424379819d478fdd093cfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499), 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68:34289)
I20250627 13:52:29.592391 730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccfdf884041949bfbfa6a09574369a13" candidate_uuid: "5c28ff2fe9424379819d478fdd093cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c4f59060974a3b823305b9dd339a7e" is_pre_election: true
I20250627 13:52:29.592451 991 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccfdf884041949bfbfa6a09574369a13" candidate_uuid: "5c28ff2fe9424379819d478fdd093cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" is_pre_election: true
I20250627 13:52:29.592620 730 raft_consensus.cc:2466] T ccfdf884041949bfbfa6a09574369a13 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c28ff2fe9424379819d478fdd093cfc in term 0.
I20250627 13:52:29.592736 991 raft_consensus.cc:2466] T ccfdf884041949bfbfa6a09574369a13 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c28ff2fe9424379819d478fdd093cfc in term 0.
I20250627 13:52:29.593080 1190 leader_election.cc:304] T ccfdf884041949bfbfa6a09574369a13 P 5c28ff2fe9424379819d478fdd093cfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c28ff2fe9424379819d478fdd093cfc, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:29.593371 1342 raft_consensus.cc:2802] T ccfdf884041949bfbfa6a09574369a13 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:29.593465 1342 raft_consensus.cc:491] T ccfdf884041949bfbfa6a09574369a13 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:29.593564 1342 raft_consensus.cc:3058] T ccfdf884041949bfbfa6a09574369a13 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:29.595384 1342 raft_consensus.cc:513] T ccfdf884041949bfbfa6a09574369a13 P 5c28ff2fe9424379819d478fdd093cfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:29.595988 1342 leader_election.cc:290] T ccfdf884041949bfbfa6a09574369a13 P 5c28ff2fe9424379819d478fdd093cfc [CANDIDATE]: Term 1 election: Requested vote from peers 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499), 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68:34289)
I20250627 13:52:29.596402 730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccfdf884041949bfbfa6a09574369a13" candidate_uuid: "5c28ff2fe9424379819d478fdd093cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c4f59060974a3b823305b9dd339a7e"
I20250627 13:52:29.596418 991 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccfdf884041949bfbfa6a09574369a13" candidate_uuid: "5c28ff2fe9424379819d478fdd093cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e8c97891d74b38b5c130d3bcbabcb6"
I20250627 13:52:29.596634 991 raft_consensus.cc:3058] T ccfdf884041949bfbfa6a09574369a13 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:29.596634 730 raft_consensus.cc:3058] T ccfdf884041949bfbfa6a09574369a13 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:29.599761 991 raft_consensus.cc:2466] T ccfdf884041949bfbfa6a09574369a13 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c28ff2fe9424379819d478fdd093cfc in term 1.
I20250627 13:52:29.599759 730 raft_consensus.cc:2466] T ccfdf884041949bfbfa6a09574369a13 P 75c4f59060974a3b823305b9dd339a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c28ff2fe9424379819d478fdd093cfc in term 1.
I20250627 13:52:29.600158 1190 leader_election.cc:304] T ccfdf884041949bfbfa6a09574369a13 P 5c28ff2fe9424379819d478fdd093cfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c28ff2fe9424379819d478fdd093cfc, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:29.600468 1342 raft_consensus.cc:2802] T ccfdf884041949bfbfa6a09574369a13 P 5c28ff2fe9424379819d478fdd093cfc [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:29.600829 1342 raft_consensus.cc:695] T ccfdf884041949bfbfa6a09574369a13 P 5c28ff2fe9424379819d478fdd093cfc [term 1 LEADER]: Becoming Leader. State: Replica: 5c28ff2fe9424379819d478fdd093cfc, State: Running, Role: LEADER
I20250627 13:52:29.601101 1342 consensus_queue.cc:237] T ccfdf884041949bfbfa6a09574369a13 P 5c28ff2fe9424379819d478fdd093cfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:29.603689 454 catalog_manager.cc:5582] T ccfdf884041949bfbfa6a09574369a13 P 5c28ff2fe9424379819d478fdd093cfc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70). New cstate: current_term: 1 leader_uuid: "5c28ff2fe9424379819d478fdd093cfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } health_report { overall_health: UNKNOWN } } }
I20250627 13:52:29.697463 1816 consensus_queue.cc:1035] T c007bb30602244e6b67ecb5970a07964 P 9f767210a91f44ae8b83b994131eee98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:29.707437 1821 consensus_queue.cc:1035] T c007bb30602244e6b67ecb5970a07964 P 9f767210a91f44ae8b83b994131eee98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:29.808300 1732 consensus_queue.cc:1035] T b77a6923c0494a5ea8471b899acff4df P 50e8c97891d74b38b5c130d3bcbabcb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:29.815258 1732 consensus_queue.cc:1035] T b77a6923c0494a5ea8471b899acff4df P 50e8c97891d74b38b5c130d3bcbabcb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250627 13:52:30.018535 1827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:30.018903 1827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:30.027173 1827 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20250627 13:52:30.065857 1342 consensus_queue.cc:1035] T ccfdf884041949bfbfa6a09574369a13 P 5c28ff2fe9424379819d478fdd093cfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:30.072423 1825 consensus_queue.cc:1035] T ccfdf884041949bfbfa6a09574369a13 P 5c28ff2fe9424379819d478fdd093cfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 287.701 ms
time per row: 0.14385 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20250627 13:52:31.452709 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49852:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250627 13:52:31.461320 1102 tablet_service.cc:1468] Processing CreateTablet for tablet 7988d92c8eff4809bfc30056446d72fd (DEFAULT_TABLE table=rebalance_test_table_6 [id=ba45580456734fd9855b703538c609cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:31.461807 1102 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7988d92c8eff4809bfc30056446d72fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:31.461769 578 tablet_service.cc:1468] Processing CreateTablet for tablet 7988d92c8eff4809bfc30056446d72fd (DEFAULT_TABLE table=rebalance_test_table_6 [id=ba45580456734fd9855b703538c609cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:31.462163 578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7988d92c8eff4809bfc30056446d72fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:31.462069 840 tablet_service.cc:1468] Processing CreateTablet for tablet 7988d92c8eff4809bfc30056446d72fd (DEFAULT_TABLE table=rebalance_test_table_6 [id=ba45580456734fd9855b703538c609cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:31.462499 840 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7988d92c8eff4809bfc30056446d72fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:31.462532 971 tablet_service.cc:1468] Processing CreateTablet for tablet fbc50222a5344df19d448cf79c899a30 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ba45580456734fd9855b703538c609cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:31.462908 971 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbc50222a5344df19d448cf79c899a30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:31.466302 1234 tablet_service.cc:1468] Processing CreateTablet for tablet fbc50222a5344df19d448cf79c899a30 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ba45580456734fd9855b703538c609cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:31.466714 1234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbc50222a5344df19d448cf79c899a30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:31.469677 710 tablet_service.cc:1468] Processing CreateTablet for tablet fbc50222a5344df19d448cf79c899a30 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ba45580456734fd9855b703538c609cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:31.470227 710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbc50222a5344df19d448cf79c899a30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:31.474180 841 tablet_service.cc:1468] Processing CreateTablet for tablet e14c8045e0374a649bf07ae94528f329 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ba45580456734fd9855b703538c609cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:31.474723 841 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e14c8045e0374a649bf07ae94528f329. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:31.477909 1898 tablet_bootstrap.cc:492] T 7988d92c8eff4809bfc30056446d72fd P 957a8a8be3f542f48e8824e581fd0f89: Bootstrap starting.
I20250627 13:52:31.478308 1102 tablet_service.cc:1468] Processing CreateTablet for tablet e14c8045e0374a649bf07ae94528f329 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ba45580456734fd9855b703538c609cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:31.478727 577 tablet_service.cc:1468] Processing CreateTablet for tablet e14c8045e0374a649bf07ae94528f329 (DEFAULT_TABLE table=rebalance_test_table_6 [id=ba45580456734fd9855b703538c609cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:31.478925 1102 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e14c8045e0374a649bf07ae94528f329. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:31.479127 577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e14c8045e0374a649bf07ae94528f329. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:31.480417 1898 tablet_bootstrap.cc:654] T 7988d92c8eff4809bfc30056446d72fd P 957a8a8be3f542f48e8824e581fd0f89: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:31.493117 1902 tablet_bootstrap.cc:492] T fbc50222a5344df19d448cf79c899a30 P 75c4f59060974a3b823305b9dd339a7e: Bootstrap starting.
I20250627 13:52:31.494114 1900 tablet_bootstrap.cc:492] T 7988d92c8eff4809bfc30056446d72fd P fa66786862da49ae826fb519c7d89b5e: Bootstrap starting.
I20250627 13:52:31.496445 1902 tablet_bootstrap.cc:654] T fbc50222a5344df19d448cf79c899a30 P 75c4f59060974a3b823305b9dd339a7e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:31.496731 1900 tablet_bootstrap.cc:654] T 7988d92c8eff4809bfc30056446d72fd P fa66786862da49ae826fb519c7d89b5e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:31.497294 1899 tablet_bootstrap.cc:492] T fbc50222a5344df19d448cf79c899a30 P 5c28ff2fe9424379819d478fdd093cfc: Bootstrap starting.
I20250627 13:52:31.498102 1897 tablet_bootstrap.cc:492] T fbc50222a5344df19d448cf79c899a30 P 50e8c97891d74b38b5c130d3bcbabcb6: Bootstrap starting.
I20250627 13:52:31.499207 1899 tablet_bootstrap.cc:654] T fbc50222a5344df19d448cf79c899a30 P 5c28ff2fe9424379819d478fdd093cfc: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:31.501737 1896 tablet_bootstrap.cc:492] T 7988d92c8eff4809bfc30056446d72fd P 9f767210a91f44ae8b83b994131eee98: Bootstrap starting.
I20250627 13:52:31.502216 1898 tablet_bootstrap.cc:492] T 7988d92c8eff4809bfc30056446d72fd P 957a8a8be3f542f48e8824e581fd0f89: No bootstrap required, opened a new log
I20250627 13:52:31.502372 1898 ts_tablet_manager.cc:1397] T 7988d92c8eff4809bfc30056446d72fd P 957a8a8be3f542f48e8824e581fd0f89: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.005s
I20250627 13:52:31.502575 1897 tablet_bootstrap.cc:654] T fbc50222a5344df19d448cf79c899a30 P 50e8c97891d74b38b5c130d3bcbabcb6: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:31.503211 1898 raft_consensus.cc:357] T 7988d92c8eff4809bfc30056446d72fd P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:31.503419 1898 raft_consensus.cc:383] T 7988d92c8eff4809bfc30056446d72fd P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:31.503515 1898 raft_consensus.cc:738] T 7988d92c8eff4809bfc30056446d72fd P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957a8a8be3f542f48e8824e581fd0f89, State: Initialized, Role: FOLLOWER
I20250627 13:52:31.503695 1896 tablet_bootstrap.cc:654] T 7988d92c8eff4809bfc30056446d72fd P 9f767210a91f44ae8b83b994131eee98: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:31.503780 1898 consensus_queue.cc:260] T 7988d92c8eff4809bfc30056446d72fd P 957a8a8be3f542f48e8824e581fd0f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:31.504316 1898 ts_tablet_manager.cc:1428] T 7988d92c8eff4809bfc30056446d72fd P 957a8a8be3f542f48e8824e581fd0f89: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250627 13:52:31.506136 1902 tablet_bootstrap.cc:492] T fbc50222a5344df19d448cf79c899a30 P 75c4f59060974a3b823305b9dd339a7e: No bootstrap required, opened a new log
I20250627 13:52:31.506302 1902 ts_tablet_manager.cc:1397] T fbc50222a5344df19d448cf79c899a30 P 75c4f59060974a3b823305b9dd339a7e: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20250627 13:52:31.507236 1899 tablet_bootstrap.cc:492] T fbc50222a5344df19d448cf79c899a30 P 5c28ff2fe9424379819d478fdd093cfc: No bootstrap required, opened a new log
I20250627 13:52:31.507424 1899 ts_tablet_manager.cc:1397] T fbc50222a5344df19d448cf79c899a30 P 5c28ff2fe9424379819d478fdd093cfc: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20250627 13:52:31.507334 1902 raft_consensus.cc:357] T fbc50222a5344df19d448cf79c899a30 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:31.507563 1902 raft_consensus.cc:383] T fbc50222a5344df19d448cf79c899a30 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:31.507743 1902 raft_consensus.cc:738] T fbc50222a5344df19d448cf79c899a30 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c4f59060974a3b823305b9dd339a7e, State: Initialized, Role: FOLLOWER
I20250627 13:52:31.508121 1902 consensus_queue.cc:260] T fbc50222a5344df19d448cf79c899a30 P 75c4f59060974a3b823305b9dd339a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:31.508286 1899 raft_consensus.cc:357] T fbc50222a5344df19d448cf79c899a30 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:31.508504 1899 raft_consensus.cc:383] T fbc50222a5344df19d448cf79c899a30 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:31.508637 1899 raft_consensus.cc:738] T fbc50222a5344df19d448cf79c899a30 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c28ff2fe9424379819d478fdd093cfc, State: Initialized, Role: FOLLOWER
I20250627 13:52:31.508873 1899 consensus_queue.cc:260] T fbc50222a5344df19d448cf79c899a30 P 5c28ff2fe9424379819d478fdd093cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:31.509449 1898 tablet_bootstrap.cc:492] T e14c8045e0374a649bf07ae94528f329 P 957a8a8be3f542f48e8824e581fd0f89: Bootstrap starting.
I20250627 13:52:31.509565 1899 ts_tablet_manager.cc:1428] T fbc50222a5344df19d448cf79c899a30 P 5c28ff2fe9424379819d478fdd093cfc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:52:31.510808 1900 tablet_bootstrap.cc:492] T 7988d92c8eff4809bfc30056446d72fd P fa66786862da49ae826fb519c7d89b5e: No bootstrap required, opened a new log
I20250627 13:52:31.511024 1900 ts_tablet_manager.cc:1397] T 7988d92c8eff4809bfc30056446d72fd P fa66786862da49ae826fb519c7d89b5e: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20250627 13:52:31.511493 1898 tablet_bootstrap.cc:654] T e14c8045e0374a649bf07ae94528f329 P 957a8a8be3f542f48e8824e581fd0f89: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:31.511897 1900 raft_consensus.cc:357] T 7988d92c8eff4809bfc30056446d72fd P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:31.512152 1900 raft_consensus.cc:383] T 7988d92c8eff4809bfc30056446d72fd P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:31.512284 1900 raft_consensus.cc:738] T 7988d92c8eff4809bfc30056446d72fd P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa66786862da49ae826fb519c7d89b5e, State: Initialized, Role: FOLLOWER
I20250627 13:52:31.512509 1902 ts_tablet_manager.cc:1428] T fbc50222a5344df19d448cf79c899a30 P 75c4f59060974a3b823305b9dd339a7e: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250627 13:52:31.512583 1900 consensus_queue.cc:260] T 7988d92c8eff4809bfc30056446d72fd P fa66786862da49ae826fb519c7d89b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:31.514297 1900 ts_tablet_manager.cc:1428] T 7988d92c8eff4809bfc30056446d72fd P fa66786862da49ae826fb519c7d89b5e: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250627 13:52:31.514628 1900 tablet_bootstrap.cc:492] T e14c8045e0374a649bf07ae94528f329 P fa66786862da49ae826fb519c7d89b5e: Bootstrap starting.
I20250627 13:52:31.515877 1900 tablet_bootstrap.cc:654] T e14c8045e0374a649bf07ae94528f329 P fa66786862da49ae826fb519c7d89b5e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:31.516577 1898 tablet_bootstrap.cc:492] T e14c8045e0374a649bf07ae94528f329 P 957a8a8be3f542f48e8824e581fd0f89: No bootstrap required, opened a new log
I20250627 13:52:31.516803 1898 ts_tablet_manager.cc:1397] T e14c8045e0374a649bf07ae94528f329 P 957a8a8be3f542f48e8824e581fd0f89: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.004s
I20250627 13:52:31.517694 1898 raft_consensus.cc:357] T e14c8045e0374a649bf07ae94528f329 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:31.517907 1898 raft_consensus.cc:383] T e14c8045e0374a649bf07ae94528f329 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:31.518039 1898 raft_consensus.cc:738] T e14c8045e0374a649bf07ae94528f329 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957a8a8be3f542f48e8824e581fd0f89, State: Initialized, Role: FOLLOWER
I20250627 13:52:31.518405 1898 consensus_queue.cc:260] T e14c8045e0374a649bf07ae94528f329 P 957a8a8be3f542f48e8824e581fd0f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:31.519104 1898 ts_tablet_manager.cc:1428] T e14c8045e0374a649bf07ae94528f329 P 957a8a8be3f542f48e8824e581fd0f89: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:52:31.521934 1897 tablet_bootstrap.cc:492] T fbc50222a5344df19d448cf79c899a30 P 50e8c97891d74b38b5c130d3bcbabcb6: No bootstrap required, opened a new log
I20250627 13:52:31.522126 1897 ts_tablet_manager.cc:1397] T fbc50222a5344df19d448cf79c899a30 P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.009s
I20250627 13:52:31.522972 1897 raft_consensus.cc:357] T fbc50222a5344df19d448cf79c899a30 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:31.523200 1897 raft_consensus.cc:383] T fbc50222a5344df19d448cf79c899a30 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:31.523329 1897 raft_consensus.cc:738] T fbc50222a5344df19d448cf79c899a30 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e8c97891d74b38b5c130d3bcbabcb6, State: Initialized, Role: FOLLOWER
I20250627 13:52:31.523627 1897 consensus_queue.cc:260] T fbc50222a5344df19d448cf79c899a30 P 50e8c97891d74b38b5c130d3bcbabcb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:31.523738 1896 tablet_bootstrap.cc:492] T 7988d92c8eff4809bfc30056446d72fd P 9f767210a91f44ae8b83b994131eee98: No bootstrap required, opened a new log
I20250627 13:52:31.524039 1896 ts_tablet_manager.cc:1397] T 7988d92c8eff4809bfc30056446d72fd P 9f767210a91f44ae8b83b994131eee98: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20250627 13:52:31.524206 1900 tablet_bootstrap.cc:492] T e14c8045e0374a649bf07ae94528f329 P fa66786862da49ae826fb519c7d89b5e: No bootstrap required, opened a new log
I20250627 13:52:31.524355 1897 ts_tablet_manager.cc:1428] T fbc50222a5344df19d448cf79c899a30 P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:52:31.524418 1900 ts_tablet_manager.cc:1397] T e14c8045e0374a649bf07ae94528f329 P fa66786862da49ae826fb519c7d89b5e: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20250627 13:52:31.524828 1896 raft_consensus.cc:357] T 7988d92c8eff4809bfc30056446d72fd P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:31.524986 1896 raft_consensus.cc:383] T 7988d92c8eff4809bfc30056446d72fd P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:31.525127 1896 raft_consensus.cc:738] T 7988d92c8eff4809bfc30056446d72fd P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f767210a91f44ae8b83b994131eee98, State: Initialized, Role: FOLLOWER
I20250627 13:52:31.525246 1900 raft_consensus.cc:357] T e14c8045e0374a649bf07ae94528f329 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:31.525337 1896 consensus_queue.cc:260] T 7988d92c8eff4809bfc30056446d72fd P 9f767210a91f44ae8b83b994131eee98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:31.525462 1900 raft_consensus.cc:383] T e14c8045e0374a649bf07ae94528f329 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:31.525578 1900 raft_consensus.cc:738] T e14c8045e0374a649bf07ae94528f329 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa66786862da49ae826fb519c7d89b5e, State: Initialized, Role: FOLLOWER
I20250627 13:52:31.525907 1896 ts_tablet_manager.cc:1428] T 7988d92c8eff4809bfc30056446d72fd P 9f767210a91f44ae8b83b994131eee98: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:52:31.525907 1900 consensus_queue.cc:260] T e14c8045e0374a649bf07ae94528f329 P fa66786862da49ae826fb519c7d89b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:31.526185 1896 tablet_bootstrap.cc:492] T e14c8045e0374a649bf07ae94528f329 P 9f767210a91f44ae8b83b994131eee98: Bootstrap starting.
I20250627 13:52:31.526391 1900 ts_tablet_manager.cc:1428] T e14c8045e0374a649bf07ae94528f329 P fa66786862da49ae826fb519c7d89b5e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250627 13:52:31.527971 1896 tablet_bootstrap.cc:654] T e14c8045e0374a649bf07ae94528f329 P 9f767210a91f44ae8b83b994131eee98: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:31.529966 1896 tablet_bootstrap.cc:492] T e14c8045e0374a649bf07ae94528f329 P 9f767210a91f44ae8b83b994131eee98: No bootstrap required, opened a new log
I20250627 13:52:31.530166 1896 ts_tablet_manager.cc:1397] T e14c8045e0374a649bf07ae94528f329 P 9f767210a91f44ae8b83b994131eee98: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250627 13:52:31.530915 1896 raft_consensus.cc:357] T e14c8045e0374a649bf07ae94528f329 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:31.531064 1896 raft_consensus.cc:383] T e14c8045e0374a649bf07ae94528f329 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:31.531143 1896 raft_consensus.cc:738] T e14c8045e0374a649bf07ae94528f329 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f767210a91f44ae8b83b994131eee98, State: Initialized, Role: FOLLOWER
I20250627 13:52:31.531359 1896 consensus_queue.cc:260] T e14c8045e0374a649bf07ae94528f329 P 9f767210a91f44ae8b83b994131eee98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:31.531782 1896 ts_tablet_manager.cc:1428] T e14c8045e0374a649bf07ae94528f329 P 9f767210a91f44ae8b83b994131eee98: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250627 13:52:31.647248 1509 raft_consensus.cc:491] T fbc50222a5344df19d448cf79c899a30 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:31.647439 1509 raft_consensus.cc:513] T fbc50222a5344df19d448cf79c899a30 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:31.648036 1509 leader_election.cc:290] T fbc50222a5344df19d448cf79c899a30 P 75c4f59060974a3b823305b9dd339a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70:39887), 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68:34289)
I20250627 13:52:31.648507 1254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fbc50222a5344df19d448cf79c899a30" candidate_uuid: "75c4f59060974a3b823305b9dd339a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c28ff2fe9424379819d478fdd093cfc" is_pre_election: true
I20250627 13:52:31.648620 991 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fbc50222a5344df19d448cf79c899a30" candidate_uuid: "75c4f59060974a3b823305b9dd339a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" is_pre_election: true
I20250627 13:52:31.648734 1254 raft_consensus.cc:2466] T fbc50222a5344df19d448cf79c899a30 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75c4f59060974a3b823305b9dd339a7e in term 0.
I20250627 13:52:31.648829 991 raft_consensus.cc:2466] T fbc50222a5344df19d448cf79c899a30 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75c4f59060974a3b823305b9dd339a7e in term 0.
I20250627 13:52:31.649185 664 leader_election.cc:304] T fbc50222a5344df19d448cf79c899a30 P 75c4f59060974a3b823305b9dd339a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c28ff2fe9424379819d478fdd093cfc, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:31.649583 1509 raft_consensus.cc:2802] T fbc50222a5344df19d448cf79c899a30 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:31.649736 1509 raft_consensus.cc:491] T fbc50222a5344df19d448cf79c899a30 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:31.649822 1509 raft_consensus.cc:3058] T fbc50222a5344df19d448cf79c899a30 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:31.651778 1509 raft_consensus.cc:513] T fbc50222a5344df19d448cf79c899a30 P 75c4f59060974a3b823305b9dd339a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:31.652201 1509 leader_election.cc:290] T fbc50222a5344df19d448cf79c899a30 P 75c4f59060974a3b823305b9dd339a7e [CANDIDATE]: Term 1 election: Requested vote from peers 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70:39887), 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68:34289)
I20250627 13:52:31.652670 1254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fbc50222a5344df19d448cf79c899a30" candidate_uuid: "75c4f59060974a3b823305b9dd339a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c28ff2fe9424379819d478fdd093cfc"
I20250627 13:52:31.652704 991 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fbc50222a5344df19d448cf79c899a30" candidate_uuid: "75c4f59060974a3b823305b9dd339a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e8c97891d74b38b5c130d3bcbabcb6"
I20250627 13:52:31.652858 1254 raft_consensus.cc:3058] T fbc50222a5344df19d448cf79c899a30 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:31.652858 991 raft_consensus.cc:3058] T fbc50222a5344df19d448cf79c899a30 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:31.655540 991 raft_consensus.cc:2466] T fbc50222a5344df19d448cf79c899a30 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75c4f59060974a3b823305b9dd339a7e in term 1.
I20250627 13:52:31.655557 1254 raft_consensus.cc:2466] T fbc50222a5344df19d448cf79c899a30 P 5c28ff2fe9424379819d478fdd093cfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75c4f59060974a3b823305b9dd339a7e in term 1.
I20250627 13:52:31.655942 665 leader_election.cc:304] T fbc50222a5344df19d448cf79c899a30 P 75c4f59060974a3b823305b9dd339a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50e8c97891d74b38b5c130d3bcbabcb6, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:31.656306 1509 raft_consensus.cc:2802] T fbc50222a5344df19d448cf79c899a30 P 75c4f59060974a3b823305b9dd339a7e [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:31.656688 1509 raft_consensus.cc:695] T fbc50222a5344df19d448cf79c899a30 P 75c4f59060974a3b823305b9dd339a7e [term 1 LEADER]: Becoming Leader. State: Replica: 75c4f59060974a3b823305b9dd339a7e, State: Running, Role: LEADER
I20250627 13:52:31.657024 1509 consensus_queue.cc:237] T fbc50222a5344df19d448cf79c899a30 P 75c4f59060974a3b823305b9dd339a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:31.659612 456 catalog_manager.cc:5582] T fbc50222a5344df19d448cf79c899a30 P 75c4f59060974a3b823305b9dd339a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to 75c4f59060974a3b823305b9dd339a7e (127.0.103.66). New cstate: current_term: 1 leader_uuid: "75c4f59060974a3b823305b9dd339a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } health_report { overall_health: HEALTHY } } }
I20250627 13:52:31.763808 1790 raft_consensus.cc:491] T e14c8045e0374a649bf07ae94528f329 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:31.764045 1790 raft_consensus.cc:513] T e14c8045e0374a649bf07ae94528f329 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:31.764720 1790 leader_election.cc:290] T e14c8045e0374a649bf07ae94528f329 P fa66786862da49ae826fb519c7d89b5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023), 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69:42325)
I20250627 13:52:31.765324 597 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e14c8045e0374a649bf07ae94528f329" candidate_uuid: "fa66786862da49ae826fb519c7d89b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98" is_pre_election: true
I20250627 13:52:31.765369 1122 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e14c8045e0374a649bf07ae94528f329" candidate_uuid: "fa66786862da49ae826fb519c7d89b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957a8a8be3f542f48e8824e581fd0f89" is_pre_election: true
I20250627 13:52:31.765560 597 raft_consensus.cc:2466] T e14c8045e0374a649bf07ae94528f329 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa66786862da49ae826fb519c7d89b5e in term 0.
I20250627 13:52:31.765614 1122 raft_consensus.cc:2466] T e14c8045e0374a649bf07ae94528f329 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa66786862da49ae826fb519c7d89b5e in term 0.
I20250627 13:52:31.766120 794 leader_election.cc:304] T e14c8045e0374a649bf07ae94528f329 P fa66786862da49ae826fb519c7d89b5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f767210a91f44ae8b83b994131eee98, fa66786862da49ae826fb519c7d89b5e; no voters:
I20250627 13:52:31.766417 1790 raft_consensus.cc:2802] T e14c8045e0374a649bf07ae94528f329 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:31.766563 1790 raft_consensus.cc:491] T e14c8045e0374a649bf07ae94528f329 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:31.766690 1790 raft_consensus.cc:3058] T e14c8045e0374a649bf07ae94528f329 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:31.768744 1790 raft_consensus.cc:513] T e14c8045e0374a649bf07ae94528f329 P fa66786862da49ae826fb519c7d89b5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:31.769290 1790 leader_election.cc:290] T e14c8045e0374a649bf07ae94528f329 P fa66786862da49ae826fb519c7d89b5e [CANDIDATE]: Term 1 election: Requested vote from peers 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023), 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69:42325)
I20250627 13:52:31.769677 597 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e14c8045e0374a649bf07ae94528f329" candidate_uuid: "fa66786862da49ae826fb519c7d89b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98"
I20250627 13:52:31.769819 1122 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e14c8045e0374a649bf07ae94528f329" candidate_uuid: "fa66786862da49ae826fb519c7d89b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957a8a8be3f542f48e8824e581fd0f89"
I20250627 13:52:31.769905 597 raft_consensus.cc:3058] T e14c8045e0374a649bf07ae94528f329 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:31.770035 1122 raft_consensus.cc:3058] T e14c8045e0374a649bf07ae94528f329 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:31.771791 597 raft_consensus.cc:2466] T e14c8045e0374a649bf07ae94528f329 P 9f767210a91f44ae8b83b994131eee98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa66786862da49ae826fb519c7d89b5e in term 1.
I20250627 13:52:31.771919 1122 raft_consensus.cc:2466] T e14c8045e0374a649bf07ae94528f329 P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa66786862da49ae826fb519c7d89b5e in term 1.
I20250627 13:52:31.772171 794 leader_election.cc:304] T e14c8045e0374a649bf07ae94528f329 P fa66786862da49ae826fb519c7d89b5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f767210a91f44ae8b83b994131eee98, fa66786862da49ae826fb519c7d89b5e; no voters:
I20250627 13:52:31.772471 1790 raft_consensus.cc:2802] T e14c8045e0374a649bf07ae94528f329 P fa66786862da49ae826fb519c7d89b5e [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:31.772856 1790 raft_consensus.cc:695] T e14c8045e0374a649bf07ae94528f329 P fa66786862da49ae826fb519c7d89b5e [term 1 LEADER]: Becoming Leader. State: Replica: fa66786862da49ae826fb519c7d89b5e, State: Running, Role: LEADER
I20250627 13:52:31.773105 1790 consensus_queue.cc:237] T e14c8045e0374a649bf07ae94528f329 P fa66786862da49ae826fb519c7d89b5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:31.775839 449 catalog_manager.cc:5582] T e14c8045e0374a649bf07ae94528f329 P fa66786862da49ae826fb519c7d89b5e reported cstate change: term changed from 0 to 1, leader changed from <none> to fa66786862da49ae826fb519c7d89b5e (127.0.103.67). New cstate: current_term: 1 leader_uuid: "fa66786862da49ae826fb519c7d89b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } health_report { overall_health: UNKNOWN } } }
I20250627 13:52:31.881632 1831 raft_consensus.cc:491] T 7988d92c8eff4809bfc30056446d72fd P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:31.881871 1831 raft_consensus.cc:513] T 7988d92c8eff4809bfc30056446d72fd P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:31.882508 1831 leader_election.cc:290] T 7988d92c8eff4809bfc30056446d72fd P 9f767210a91f44ae8b83b994131eee98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69:42325), fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747)
I20250627 13:52:31.883009 1122 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7988d92c8eff4809bfc30056446d72fd" candidate_uuid: "9f767210a91f44ae8b83b994131eee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957a8a8be3f542f48e8824e581fd0f89" is_pre_election: true
I20250627 13:52:31.883026 861 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7988d92c8eff4809bfc30056446d72fd" candidate_uuid: "9f767210a91f44ae8b83b994131eee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e" is_pre_election: true
I20250627 13:52:31.883225 861 raft_consensus.cc:2466] T 7988d92c8eff4809bfc30056446d72fd P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f767210a91f44ae8b83b994131eee98 in term 0.
I20250627 13:52:31.883244 1122 raft_consensus.cc:2466] T 7988d92c8eff4809bfc30056446d72fd P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f767210a91f44ae8b83b994131eee98 in term 0.
I20250627 13:52:31.883639 534 leader_election.cc:304] T 7988d92c8eff4809bfc30056446d72fd P 9f767210a91f44ae8b83b994131eee98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, 9f767210a91f44ae8b83b994131eee98; no voters:
I20250627 13:52:31.883911 1831 raft_consensus.cc:2802] T 7988d92c8eff4809bfc30056446d72fd P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:31.884008 1831 raft_consensus.cc:491] T 7988d92c8eff4809bfc30056446d72fd P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:31.884100 1831 raft_consensus.cc:3058] T 7988d92c8eff4809bfc30056446d72fd P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:31.886020 1831 raft_consensus.cc:513] T 7988d92c8eff4809bfc30056446d72fd P 9f767210a91f44ae8b83b994131eee98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:31.886505 1831 leader_election.cc:290] T 7988d92c8eff4809bfc30056446d72fd P 9f767210a91f44ae8b83b994131eee98 [CANDIDATE]: Term 1 election: Requested vote from peers 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69:42325), fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747)
I20250627 13:52:31.886888 1122 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7988d92c8eff4809bfc30056446d72fd" candidate_uuid: "9f767210a91f44ae8b83b994131eee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957a8a8be3f542f48e8824e581fd0f89"
I20250627 13:52:31.886914 861 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7988d92c8eff4809bfc30056446d72fd" candidate_uuid: "9f767210a91f44ae8b83b994131eee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e"
I20250627 13:52:31.887089 861 raft_consensus.cc:3058] T 7988d92c8eff4809bfc30056446d72fd P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:31.887079 1122 raft_consensus.cc:3058] T 7988d92c8eff4809bfc30056446d72fd P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:31.889534 861 raft_consensus.cc:2466] T 7988d92c8eff4809bfc30056446d72fd P fa66786862da49ae826fb519c7d89b5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f767210a91f44ae8b83b994131eee98 in term 1.
I20250627 13:52:31.889627 1122 raft_consensus.cc:2466] T 7988d92c8eff4809bfc30056446d72fd P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f767210a91f44ae8b83b994131eee98 in term 1.
I20250627 13:52:31.889910 533 leader_election.cc:304] T 7988d92c8eff4809bfc30056446d72fd P 9f767210a91f44ae8b83b994131eee98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f767210a91f44ae8b83b994131eee98, fa66786862da49ae826fb519c7d89b5e; no voters:
I20250627 13:52:31.890197 1831 raft_consensus.cc:2802] T 7988d92c8eff4809bfc30056446d72fd P 9f767210a91f44ae8b83b994131eee98 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:31.890527 1831 raft_consensus.cc:695] T 7988d92c8eff4809bfc30056446d72fd P 9f767210a91f44ae8b83b994131eee98 [term 1 LEADER]: Becoming Leader. State: Replica: 9f767210a91f44ae8b83b994131eee98, State: Running, Role: LEADER
I20250627 13:52:31.890776 1831 consensus_queue.cc:237] T 7988d92c8eff4809bfc30056446d72fd P 9f767210a91f44ae8b83b994131eee98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:31.893326 449 catalog_manager.cc:5582] T 7988d92c8eff4809bfc30056446d72fd P 9f767210a91f44ae8b83b994131eee98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f767210a91f44ae8b83b994131eee98 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "9f767210a91f44ae8b83b994131eee98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } health_report { overall_health: HEALTHY } } }
I20250627 13:52:32.081977 1509 consensus_queue.cc:1035] T fbc50222a5344df19d448cf79c899a30 P 75c4f59060974a3b823305b9dd339a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:32.087802 1509 consensus_queue.cc:1035] T fbc50222a5344df19d448cf79c899a30 P 75c4f59060974a3b823305b9dd339a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:32.170042 1913 consensus_queue.cc:1035] T e14c8045e0374a649bf07ae94528f329 P fa66786862da49ae826fb519c7d89b5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:32.179730 1926 consensus_queue.cc:1035] T e14c8045e0374a649bf07ae94528f329 P fa66786862da49ae826fb519c7d89b5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250627 13:52:32.266767 1917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:32.267220 1917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:32.276403 1917 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20250627 13:52:32.383649 1831 consensus_queue.cc:1035] T 7988d92c8eff4809bfc30056446d72fd P 9f767210a91f44ae8b83b994131eee98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:32.415009 1831 consensus_queue.cc:1035] T 7988d92c8eff4809bfc30056446d72fd P 9f767210a91f44ae8b83b994131eee98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 307.408 ms
time per row: 0.153704 ms
I20250627 13:52:32.738906 456 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49852:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250627 13:52:32.748400 710 tablet_service.cc:1468] Processing CreateTablet for tablet 8ced07e8b66d4b37a317e28713fe518d (DEFAULT_TABLE table=rebalance_test_table_7 [id=d33a3287ac8a4b72a44966c18c5e92c5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:32.748879 710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ced07e8b66d4b37a317e28713fe518d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:32.748960 1234 tablet_service.cc:1468] Processing CreateTablet for tablet 8ced07e8b66d4b37a317e28713fe518d (DEFAULT_TABLE table=rebalance_test_table_7 [id=d33a3287ac8a4b72a44966c18c5e92c5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:32.749397 1234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ced07e8b66d4b37a317e28713fe518d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:32.749919 972 tablet_service.cc:1468] Processing CreateTablet for tablet d146d21a389944a0b47e89e85a6fe920 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d33a3287ac8a4b72a44966c18c5e92c5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:32.750305 972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d146d21a389944a0b47e89e85a6fe920. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:32.753844 1986 tablet_bootstrap.cc:492] T 8ced07e8b66d4b37a317e28713fe518d P 75c4f59060974a3b823305b9dd339a7e: Bootstrap starting.
I20250627 13:52:32.753795 709 tablet_service.cc:1468] Processing CreateTablet for tablet d146d21a389944a0b47e89e85a6fe920 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d33a3287ac8a4b72a44966c18c5e92c5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:32.754196 709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d146d21a389944a0b47e89e85a6fe920. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:32.755897 1986 tablet_bootstrap.cc:654] T 8ced07e8b66d4b37a317e28713fe518d P 75c4f59060974a3b823305b9dd339a7e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:32.755947 971 tablet_service.cc:1468] Processing CreateTablet for tablet 8ced07e8b66d4b37a317e28713fe518d (DEFAULT_TABLE table=rebalance_test_table_7 [id=d33a3287ac8a4b72a44966c18c5e92c5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:32.756327 971 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ced07e8b66d4b37a317e28713fe518d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:32.756851 840 tablet_service.cc:1468] Processing CreateTablet for tablet 012c8298a5d741909d6838470e8c05ba (DEFAULT_TABLE table=rebalance_test_table_7 [id=d33a3287ac8a4b72a44966c18c5e92c5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:32.757222 840 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 012c8298a5d741909d6838470e8c05ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:32.758193 578 tablet_service.cc:1468] Processing CreateTablet for tablet 012c8298a5d741909d6838470e8c05ba (DEFAULT_TABLE table=rebalance_test_table_7 [id=d33a3287ac8a4b72a44966c18c5e92c5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:32.758703 578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 012c8298a5d741909d6838470e8c05ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:32.763173 1233 tablet_service.cc:1468] Processing CreateTablet for tablet d146d21a389944a0b47e89e85a6fe920 (DEFAULT_TABLE table=rebalance_test_table_7 [id=d33a3287ac8a4b72a44966c18c5e92c5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:32.763696 1233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d146d21a389944a0b47e89e85a6fe920. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:32.764273 1102 tablet_service.cc:1468] Processing CreateTablet for tablet 012c8298a5d741909d6838470e8c05ba (DEFAULT_TABLE table=rebalance_test_table_7 [id=d33a3287ac8a4b72a44966c18c5e92c5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:32.764665 1102 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 012c8298a5d741909d6838470e8c05ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:32.786556 1992 tablet_bootstrap.cc:492] T 012c8298a5d741909d6838470e8c05ba P 9f767210a91f44ae8b83b994131eee98: Bootstrap starting.
I20250627 13:52:32.789129 1992 tablet_bootstrap.cc:654] T 012c8298a5d741909d6838470e8c05ba P 9f767210a91f44ae8b83b994131eee98: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:32.791985 1991 tablet_bootstrap.cc:492] T 012c8298a5d741909d6838470e8c05ba P fa66786862da49ae826fb519c7d89b5e: Bootstrap starting.
I20250627 13:52:32.792747 1988 tablet_bootstrap.cc:492] T d146d21a389944a0b47e89e85a6fe920 P 50e8c97891d74b38b5c130d3bcbabcb6: Bootstrap starting.
I20250627 13:52:32.794386 1990 tablet_bootstrap.cc:492] T 012c8298a5d741909d6838470e8c05ba P 957a8a8be3f542f48e8824e581fd0f89: Bootstrap starting.
I20250627 13:52:32.795293 1991 tablet_bootstrap.cc:654] T 012c8298a5d741909d6838470e8c05ba P fa66786862da49ae826fb519c7d89b5e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:32.795917 1987 tablet_bootstrap.cc:492] T 8ced07e8b66d4b37a317e28713fe518d P 5c28ff2fe9424379819d478fdd093cfc: Bootstrap starting.
I20250627 13:52:32.797248 1990 tablet_bootstrap.cc:654] T 012c8298a5d741909d6838470e8c05ba P 957a8a8be3f542f48e8824e581fd0f89: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:32.797616 1988 tablet_bootstrap.cc:654] T d146d21a389944a0b47e89e85a6fe920 P 50e8c97891d74b38b5c130d3bcbabcb6: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:32.798202 1987 tablet_bootstrap.cc:654] T 8ced07e8b66d4b37a317e28713fe518d P 5c28ff2fe9424379819d478fdd093cfc: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:32.808215 1987 tablet_bootstrap.cc:492] T 8ced07e8b66d4b37a317e28713fe518d P 5c28ff2fe9424379819d478fdd093cfc: No bootstrap required, opened a new log
I20250627 13:52:32.808503 1987 ts_tablet_manager.cc:1397] T 8ced07e8b66d4b37a317e28713fe518d P 5c28ff2fe9424379819d478fdd093cfc: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20250627 13:52:32.810755 1986 tablet_bootstrap.cc:492] T 8ced07e8b66d4b37a317e28713fe518d P 75c4f59060974a3b823305b9dd339a7e: No bootstrap required, opened a new log
I20250627 13:52:32.810804 1987 raft_consensus.cc:357] T 8ced07e8b66d4b37a317e28713fe518d P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:32.811043 1986 ts_tablet_manager.cc:1397] T 8ced07e8b66d4b37a317e28713fe518d P 75c4f59060974a3b823305b9dd339a7e: Time spent bootstrapping tablet: real 0.057s user 0.017s sys 0.021s
I20250627 13:52:32.811180 1987 raft_consensus.cc:383] T 8ced07e8b66d4b37a317e28713fe518d P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:32.811360 1987 raft_consensus.cc:738] T 8ced07e8b66d4b37a317e28713fe518d P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c28ff2fe9424379819d478fdd093cfc, State: Initialized, Role: FOLLOWER
I20250627 13:52:32.811748 1987 consensus_queue.cc:260] T 8ced07e8b66d4b37a317e28713fe518d P 5c28ff2fe9424379819d478fdd093cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:32.812041 1986 raft_consensus.cc:357] T 8ced07e8b66d4b37a317e28713fe518d P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:32.812322 1986 raft_consensus.cc:383] T 8ced07e8b66d4b37a317e28713fe518d P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:32.812453 1986 raft_consensus.cc:738] T 8ced07e8b66d4b37a317e28713fe518d P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c4f59060974a3b823305b9dd339a7e, State: Initialized, Role: FOLLOWER
I20250627 13:52:32.812657 1987 ts_tablet_manager.cc:1428] T 8ced07e8b66d4b37a317e28713fe518d P 5c28ff2fe9424379819d478fdd093cfc: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250627 13:52:32.812754 1986 consensus_queue.cc:260] T 8ced07e8b66d4b37a317e28713fe518d P 75c4f59060974a3b823305b9dd339a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:32.813109 1987 tablet_bootstrap.cc:492] T d146d21a389944a0b47e89e85a6fe920 P 5c28ff2fe9424379819d478fdd093cfc: Bootstrap starting.
I20250627 13:52:32.815057 1986 ts_tablet_manager.cc:1428] T 8ced07e8b66d4b37a317e28713fe518d P 75c4f59060974a3b823305b9dd339a7e: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250627 13:52:32.815413 1986 tablet_bootstrap.cc:492] T d146d21a389944a0b47e89e85a6fe920 P 75c4f59060974a3b823305b9dd339a7e: Bootstrap starting.
I20250627 13:52:32.815696 1987 tablet_bootstrap.cc:654] T d146d21a389944a0b47e89e85a6fe920 P 5c28ff2fe9424379819d478fdd093cfc: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:32.817094 1986 tablet_bootstrap.cc:654] T d146d21a389944a0b47e89e85a6fe920 P 75c4f59060974a3b823305b9dd339a7e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:32.822305 1992 tablet_bootstrap.cc:492] T 012c8298a5d741909d6838470e8c05ba P 9f767210a91f44ae8b83b994131eee98: No bootstrap required, opened a new log
I20250627 13:52:32.822495 1992 ts_tablet_manager.cc:1397] T 012c8298a5d741909d6838470e8c05ba P 9f767210a91f44ae8b83b994131eee98: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.004s
I20250627 13:52:32.823343 1992 raft_consensus.cc:357] T 012c8298a5d741909d6838470e8c05ba P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:32.823761 1992 raft_consensus.cc:383] T 012c8298a5d741909d6838470e8c05ba P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:32.824105 1987 tablet_bootstrap.cc:492] T d146d21a389944a0b47e89e85a6fe920 P 5c28ff2fe9424379819d478fdd093cfc: No bootstrap required, opened a new log
I20250627 13:52:32.824127 1990 tablet_bootstrap.cc:492] T 012c8298a5d741909d6838470e8c05ba P 957a8a8be3f542f48e8824e581fd0f89: No bootstrap required, opened a new log
I20250627 13:52:32.824265 1990 ts_tablet_manager.cc:1397] T 012c8298a5d741909d6838470e8c05ba P 957a8a8be3f542f48e8824e581fd0f89: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.015s
I20250627 13:52:32.824306 1987 ts_tablet_manager.cc:1397] T d146d21a389944a0b47e89e85a6fe920 P 5c28ff2fe9424379819d478fdd093cfc: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20250627 13:52:32.824549 1991 tablet_bootstrap.cc:492] T 012c8298a5d741909d6838470e8c05ba P fa66786862da49ae826fb519c7d89b5e: No bootstrap required, opened a new log
I20250627 13:52:32.824771 1991 ts_tablet_manager.cc:1397] T 012c8298a5d741909d6838470e8c05ba P fa66786862da49ae826fb519c7d89b5e: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.005s
I20250627 13:52:32.825044 1987 raft_consensus.cc:357] T d146d21a389944a0b47e89e85a6fe920 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:32.825074 1990 raft_consensus.cc:357] T 012c8298a5d741909d6838470e8c05ba P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:32.825232 1990 raft_consensus.cc:383] T 012c8298a5d741909d6838470e8c05ba P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:32.825284 1987 raft_consensus.cc:383] T d146d21a389944a0b47e89e85a6fe920 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:32.825304 1992 raft_consensus.cc:738] T 012c8298a5d741909d6838470e8c05ba P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f767210a91f44ae8b83b994131eee98, State: Initialized, Role: FOLLOWER
I20250627 13:52:32.825359 1990 raft_consensus.cc:738] T 012c8298a5d741909d6838470e8c05ba P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957a8a8be3f542f48e8824e581fd0f89, State: Initialized, Role: FOLLOWER
I20250627 13:52:32.825388 1987 raft_consensus.cc:738] T d146d21a389944a0b47e89e85a6fe920 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c28ff2fe9424379819d478fdd093cfc, State: Initialized, Role: FOLLOWER
I20250627 13:52:32.825619 1990 consensus_queue.cc:260] T 012c8298a5d741909d6838470e8c05ba P 957a8a8be3f542f48e8824e581fd0f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:32.825686 1987 consensus_queue.cc:260] T d146d21a389944a0b47e89e85a6fe920 P 5c28ff2fe9424379819d478fdd093cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:32.825726 1992 consensus_queue.cc:260] T 012c8298a5d741909d6838470e8c05ba P 9f767210a91f44ae8b83b994131eee98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:32.826143 1987 ts_tablet_manager.cc:1428] T d146d21a389944a0b47e89e85a6fe920 P 5c28ff2fe9424379819d478fdd093cfc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:52:32.827163 1988 tablet_bootstrap.cc:492] T d146d21a389944a0b47e89e85a6fe920 P 50e8c97891d74b38b5c130d3bcbabcb6: No bootstrap required, opened a new log
I20250627 13:52:32.827338 1988 ts_tablet_manager.cc:1397] T d146d21a389944a0b47e89e85a6fe920 P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.004s
I20250627 13:52:32.827436 1990 ts_tablet_manager.cc:1428] T 012c8298a5d741909d6838470e8c05ba P 957a8a8be3f542f48e8824e581fd0f89: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250627 13:52:32.828158 1988 raft_consensus.cc:357] T d146d21a389944a0b47e89e85a6fe920 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:32.828501 1988 raft_consensus.cc:383] T d146d21a389944a0b47e89e85a6fe920 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:32.828662 1988 raft_consensus.cc:738] T d146d21a389944a0b47e89e85a6fe920 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e8c97891d74b38b5c130d3bcbabcb6, State: Initialized, Role: FOLLOWER
I20250627 13:52:32.828966 1988 consensus_queue.cc:260] T d146d21a389944a0b47e89e85a6fe920 P 50e8c97891d74b38b5c130d3bcbabcb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:32.830763 1988 ts_tablet_manager.cc:1428] T d146d21a389944a0b47e89e85a6fe920 P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250627 13:52:32.831071 1988 tablet_bootstrap.cc:492] T 8ced07e8b66d4b37a317e28713fe518d P 50e8c97891d74b38b5c130d3bcbabcb6: Bootstrap starting.
I20250627 13:52:32.826335 1991 raft_consensus.cc:357] T 012c8298a5d741909d6838470e8c05ba P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:32.833317 1992 ts_tablet_manager.cc:1428] T 012c8298a5d741909d6838470e8c05ba P 9f767210a91f44ae8b83b994131eee98: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250627 13:52:32.833465 1991 raft_consensus.cc:383] T 012c8298a5d741909d6838470e8c05ba P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:32.833635 1988 tablet_bootstrap.cc:654] T 8ced07e8b66d4b37a317e28713fe518d P 50e8c97891d74b38b5c130d3bcbabcb6: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:32.833791 1991 raft_consensus.cc:738] T 012c8298a5d741909d6838470e8c05ba P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa66786862da49ae826fb519c7d89b5e, State: Initialized, Role: FOLLOWER
I20250627 13:52:32.834734 1986 tablet_bootstrap.cc:492] T d146d21a389944a0b47e89e85a6fe920 P 75c4f59060974a3b823305b9dd339a7e: No bootstrap required, opened a new log
I20250627 13:52:32.834914 1986 ts_tablet_manager.cc:1397] T d146d21a389944a0b47e89e85a6fe920 P 75c4f59060974a3b823305b9dd339a7e: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.005s
I20250627 13:52:32.836166 1986 raft_consensus.cc:357] T d146d21a389944a0b47e89e85a6fe920 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:32.836375 1986 raft_consensus.cc:383] T d146d21a389944a0b47e89e85a6fe920 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:32.836458 1986 raft_consensus.cc:738] T d146d21a389944a0b47e89e85a6fe920 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c4f59060974a3b823305b9dd339a7e, State: Initialized, Role: FOLLOWER
I20250627 13:52:32.836228 1991 consensus_queue.cc:260] T 012c8298a5d741909d6838470e8c05ba P fa66786862da49ae826fb519c7d89b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:32.836704 1986 consensus_queue.cc:260] T d146d21a389944a0b47e89e85a6fe920 P 75c4f59060974a3b823305b9dd339a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:32.837193 1986 ts_tablet_manager.cc:1428] T d146d21a389944a0b47e89e85a6fe920 P 75c4f59060974a3b823305b9dd339a7e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250627 13:52:32.840401 1991 ts_tablet_manager.cc:1428] T 012c8298a5d741909d6838470e8c05ba P fa66786862da49ae826fb519c7d89b5e: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250627 13:52:32.841158 1988 tablet_bootstrap.cc:492] T 8ced07e8b66d4b37a317e28713fe518d P 50e8c97891d74b38b5c130d3bcbabcb6: No bootstrap required, opened a new log
I20250627 13:52:32.842005 1988 ts_tablet_manager.cc:1397] T 8ced07e8b66d4b37a317e28713fe518d P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20250627 13:52:32.842796 1988 raft_consensus.cc:357] T 8ced07e8b66d4b37a317e28713fe518d P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:32.842995 1988 raft_consensus.cc:383] T 8ced07e8b66d4b37a317e28713fe518d P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:32.843108 1988 raft_consensus.cc:738] T 8ced07e8b66d4b37a317e28713fe518d P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e8c97891d74b38b5c130d3bcbabcb6, State: Initialized, Role: FOLLOWER
I20250627 13:52:32.843391 1988 consensus_queue.cc:260] T 8ced07e8b66d4b37a317e28713fe518d P 50e8c97891d74b38b5c130d3bcbabcb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:32.844604 1988 ts_tablet_manager.cc:1428] T 8ced07e8b66d4b37a317e28713fe518d P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:52:32.857192 1509 raft_consensus.cc:491] T 8ced07e8b66d4b37a317e28713fe518d P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:32.857386 1509 raft_consensus.cc:513] T 8ced07e8b66d4b37a317e28713fe518d P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:32.858071 1509 leader_election.cc:290] T 8ced07e8b66d4b37a317e28713fe518d P 75c4f59060974a3b823305b9dd339a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70:39887), 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68:34289)
I20250627 13:52:32.858613 1253 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ced07e8b66d4b37a317e28713fe518d" candidate_uuid: "75c4f59060974a3b823305b9dd339a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c28ff2fe9424379819d478fdd093cfc" is_pre_election: true
I20250627 13:52:32.858768 991 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ced07e8b66d4b37a317e28713fe518d" candidate_uuid: "75c4f59060974a3b823305b9dd339a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" is_pre_election: true
I20250627 13:52:32.858825 1253 raft_consensus.cc:2466] T 8ced07e8b66d4b37a317e28713fe518d P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75c4f59060974a3b823305b9dd339a7e in term 0.
I20250627 13:52:32.859007 991 raft_consensus.cc:2466] T 8ced07e8b66d4b37a317e28713fe518d P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75c4f59060974a3b823305b9dd339a7e in term 0.
I20250627 13:52:32.859234 664 leader_election.cc:304] T 8ced07e8b66d4b37a317e28713fe518d P 75c4f59060974a3b823305b9dd339a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c28ff2fe9424379819d478fdd093cfc, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:32.859527 1509 raft_consensus.cc:2802] T 8ced07e8b66d4b37a317e28713fe518d P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:32.859627 1509 raft_consensus.cc:491] T 8ced07e8b66d4b37a317e28713fe518d P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:32.859732 1509 raft_consensus.cc:3058] T 8ced07e8b66d4b37a317e28713fe518d P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:32.861572 1509 raft_consensus.cc:513] T 8ced07e8b66d4b37a317e28713fe518d P 75c4f59060974a3b823305b9dd339a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:32.862150 1509 leader_election.cc:290] T 8ced07e8b66d4b37a317e28713fe518d P 75c4f59060974a3b823305b9dd339a7e [CANDIDATE]: Term 1 election: Requested vote from peers 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70:39887), 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68:34289)
I20250627 13:52:32.862450 1253 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ced07e8b66d4b37a317e28713fe518d" candidate_uuid: "75c4f59060974a3b823305b9dd339a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c28ff2fe9424379819d478fdd093cfc"
I20250627 13:52:32.862577 991 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ced07e8b66d4b37a317e28713fe518d" candidate_uuid: "75c4f59060974a3b823305b9dd339a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e8c97891d74b38b5c130d3bcbabcb6"
I20250627 13:52:32.862638 1253 raft_consensus.cc:3058] T 8ced07e8b66d4b37a317e28713fe518d P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:32.862771 991 raft_consensus.cc:3058] T 8ced07e8b66d4b37a317e28713fe518d P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:32.862895 1890 raft_consensus.cc:491] T 012c8298a5d741909d6838470e8c05ba P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:32.863031 1890 raft_consensus.cc:513] T 012c8298a5d741909d6838470e8c05ba P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:32.863597 1890 leader_election.cc:290] T 012c8298a5d741909d6838470e8c05ba P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747), 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023)
I20250627 13:52:32.864181 860 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "012c8298a5d741909d6838470e8c05ba" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e" is_pre_election: true
I20250627 13:52:32.864374 860 raft_consensus.cc:2466] T 012c8298a5d741909d6838470e8c05ba P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 0.
I20250627 13:52:32.865016 598 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "012c8298a5d741909d6838470e8c05ba" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98" is_pre_election: true
I20250627 13:52:32.865198 598 raft_consensus.cc:2466] T 012c8298a5d741909d6838470e8c05ba P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 0.
I20250627 13:52:32.865427 1253 raft_consensus.cc:2466] T 8ced07e8b66d4b37a317e28713fe518d P 5c28ff2fe9424379819d478fdd093cfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75c4f59060974a3b823305b9dd339a7e in term 1.
I20250627 13:52:32.865443 991 raft_consensus.cc:2466] T 8ced07e8b66d4b37a317e28713fe518d P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75c4f59060974a3b823305b9dd339a7e in term 1.
I20250627 13:52:32.865747 1058 leader_election.cc:304] T 012c8298a5d741909d6838470e8c05ba P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, fa66786862da49ae826fb519c7d89b5e; no voters:
I20250627 13:52:32.865795 664 leader_election.cc:304] T 8ced07e8b66d4b37a317e28713fe518d P 75c4f59060974a3b823305b9dd339a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c28ff2fe9424379819d478fdd093cfc, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:32.866086 1509 raft_consensus.cc:2802] T 8ced07e8b66d4b37a317e28713fe518d P 75c4f59060974a3b823305b9dd339a7e [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:32.866133 1890 raft_consensus.cc:2802] T 012c8298a5d741909d6838470e8c05ba P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:32.866225 1890 raft_consensus.cc:491] T 012c8298a5d741909d6838470e8c05ba P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:32.866297 1890 raft_consensus.cc:3058] T 012c8298a5d741909d6838470e8c05ba P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:32.866567 1509 raft_consensus.cc:695] T 8ced07e8b66d4b37a317e28713fe518d P 75c4f59060974a3b823305b9dd339a7e [term 1 LEADER]: Becoming Leader. State: Replica: 75c4f59060974a3b823305b9dd339a7e, State: Running, Role: LEADER
I20250627 13:52:32.866811 1509 consensus_queue.cc:237] T 8ced07e8b66d4b37a317e28713fe518d P 75c4f59060974a3b823305b9dd339a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:32.869205 1890 raft_consensus.cc:513] T 012c8298a5d741909d6838470e8c05ba P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:32.870158 860 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "012c8298a5d741909d6838470e8c05ba" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e"
I20250627 13:52:32.870384 860 raft_consensus.cc:3058] T 012c8298a5d741909d6838470e8c05ba P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:32.870157 451 catalog_manager.cc:5582] T 8ced07e8b66d4b37a317e28713fe518d P 75c4f59060974a3b823305b9dd339a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to 75c4f59060974a3b823305b9dd339a7e (127.0.103.66). New cstate: current_term: 1 leader_uuid: "75c4f59060974a3b823305b9dd339a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } health_report { overall_health: UNKNOWN } } }
I20250627 13:52:32.873152 860 raft_consensus.cc:2466] T 012c8298a5d741909d6838470e8c05ba P fa66786862da49ae826fb519c7d89b5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 1.
I20250627 13:52:32.873586 1058 leader_election.cc:304] T 012c8298a5d741909d6838470e8c05ba P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, fa66786862da49ae826fb519c7d89b5e; no voters:
I20250627 13:52:32.874686 598 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "012c8298a5d741909d6838470e8c05ba" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98"
I20250627 13:52:32.874902 598 raft_consensus.cc:3058] T 012c8298a5d741909d6838470e8c05ba P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:32.876833 598 raft_consensus.cc:2466] T 012c8298a5d741909d6838470e8c05ba P 9f767210a91f44ae8b83b994131eee98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 1.
I20250627 13:52:32.877219 1890 leader_election.cc:290] T 012c8298a5d741909d6838470e8c05ba P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 election: Requested vote from peers fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747), 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023)
I20250627 13:52:32.877429 1890 raft_consensus.cc:2802] T 012c8298a5d741909d6838470e8c05ba P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:32.877545 1890 raft_consensus.cc:695] T 012c8298a5d741909d6838470e8c05ba P 957a8a8be3f542f48e8824e581fd0f89 [term 1 LEADER]: Becoming Leader. State: Replica: 957a8a8be3f542f48e8824e581fd0f89, State: Running, Role: LEADER
I20250627 13:52:32.877810 1890 consensus_queue.cc:237] T 012c8298a5d741909d6838470e8c05ba P 957a8a8be3f542f48e8824e581fd0f89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:32.880700 451 catalog_manager.cc:5582] T 012c8298a5d741909d6838470e8c05ba P 957a8a8be3f542f48e8824e581fd0f89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69). New cstate: current_term: 1 leader_uuid: "957a8a8be3f542f48e8824e581fd0f89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } health_report { overall_health: HEALTHY } } }
I20250627 13:52:32.887096 1929 raft_consensus.cc:491] T d146d21a389944a0b47e89e85a6fe920 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:32.887249 1929 raft_consensus.cc:513] T d146d21a389944a0b47e89e85a6fe920 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:32.887807 1929 leader_election.cc:290] T d146d21a389944a0b47e89e85a6fe920 P 5c28ff2fe9424379819d478fdd093cfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499), 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68:34289)
I20250627 13:52:32.888255 991 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d146d21a389944a0b47e89e85a6fe920" candidate_uuid: "5c28ff2fe9424379819d478fdd093cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" is_pre_election: true
I20250627 13:52:32.888268 730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d146d21a389944a0b47e89e85a6fe920" candidate_uuid: "5c28ff2fe9424379819d478fdd093cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c4f59060974a3b823305b9dd339a7e" is_pre_election: true
I20250627 13:52:32.888430 991 raft_consensus.cc:2466] T d146d21a389944a0b47e89e85a6fe920 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c28ff2fe9424379819d478fdd093cfc in term 0.
I20250627 13:52:32.888447 730 raft_consensus.cc:2466] T d146d21a389944a0b47e89e85a6fe920 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c28ff2fe9424379819d478fdd093cfc in term 0.
I20250627 13:52:32.888833 1189 leader_election.cc:304] T d146d21a389944a0b47e89e85a6fe920 P 5c28ff2fe9424379819d478fdd093cfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50e8c97891d74b38b5c130d3bcbabcb6, 5c28ff2fe9424379819d478fdd093cfc; no voters:
I20250627 13:52:32.889092 1929 raft_consensus.cc:2802] T d146d21a389944a0b47e89e85a6fe920 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:32.889230 1929 raft_consensus.cc:491] T d146d21a389944a0b47e89e85a6fe920 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:32.889317 1929 raft_consensus.cc:3058] T d146d21a389944a0b47e89e85a6fe920 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:32.891180 1929 raft_consensus.cc:513] T d146d21a389944a0b47e89e85a6fe920 P 5c28ff2fe9424379819d478fdd093cfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:32.891644 1929 leader_election.cc:290] T d146d21a389944a0b47e89e85a6fe920 P 5c28ff2fe9424379819d478fdd093cfc [CANDIDATE]: Term 1 election: Requested vote from peers 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499), 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68:34289)
I20250627 13:52:32.892124 730 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d146d21a389944a0b47e89e85a6fe920" candidate_uuid: "5c28ff2fe9424379819d478fdd093cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c4f59060974a3b823305b9dd339a7e"
I20250627 13:52:32.892138 991 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d146d21a389944a0b47e89e85a6fe920" candidate_uuid: "5c28ff2fe9424379819d478fdd093cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e8c97891d74b38b5c130d3bcbabcb6"
I20250627 13:52:32.892310 991 raft_consensus.cc:3058] T d146d21a389944a0b47e89e85a6fe920 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:32.892310 730 raft_consensus.cc:3058] T d146d21a389944a0b47e89e85a6fe920 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:32.895387 991 raft_consensus.cc:2466] T d146d21a389944a0b47e89e85a6fe920 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c28ff2fe9424379819d478fdd093cfc in term 1.
I20250627 13:52:32.895387 730 raft_consensus.cc:2466] T d146d21a389944a0b47e89e85a6fe920 P 75c4f59060974a3b823305b9dd339a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c28ff2fe9424379819d478fdd093cfc in term 1.
I20250627 13:52:32.895766 1189 leader_election.cc:304] T d146d21a389944a0b47e89e85a6fe920 P 5c28ff2fe9424379819d478fdd093cfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50e8c97891d74b38b5c130d3bcbabcb6, 5c28ff2fe9424379819d478fdd093cfc; no voters:
I20250627 13:52:32.896065 1929 raft_consensus.cc:2802] T d146d21a389944a0b47e89e85a6fe920 P 5c28ff2fe9424379819d478fdd093cfc [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:32.896431 1929 raft_consensus.cc:695] T d146d21a389944a0b47e89e85a6fe920 P 5c28ff2fe9424379819d478fdd093cfc [term 1 LEADER]: Becoming Leader. State: Replica: 5c28ff2fe9424379819d478fdd093cfc, State: Running, Role: LEADER
I20250627 13:52:32.896732 1929 consensus_queue.cc:237] T d146d21a389944a0b47e89e85a6fe920 P 5c28ff2fe9424379819d478fdd093cfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:32.899401 451 catalog_manager.cc:5582] T d146d21a389944a0b47e89e85a6fe920 P 5c28ff2fe9424379819d478fdd093cfc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70). New cstate: current_term: 1 leader_uuid: "5c28ff2fe9424379819d478fdd093cfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } health_report { overall_health: UNKNOWN } } }
W20250627 13:52:33.245615 2006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:33.246006 2006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:33.254619 2006 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20250627 13:52:33.307840 1509 consensus_queue.cc:1035] T 8ced07e8b66d4b37a317e28713fe518d P 75c4f59060974a3b823305b9dd339a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:33.331735 1911 consensus_queue.cc:1035] T 8ced07e8b66d4b37a317e28713fe518d P 75c4f59060974a3b823305b9dd339a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:33.349153 1929 consensus_queue.cc:1035] T d146d21a389944a0b47e89e85a6fe920 P 5c28ff2fe9424379819d478fdd093cfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:33.366044 2004 consensus_queue.cc:1035] T d146d21a389944a0b47e89e85a6fe920 P 5c28ff2fe9424379819d478fdd093cfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:33.375797 1890 consensus_queue.cc:1035] T 012c8298a5d741909d6838470e8c05ba P 957a8a8be3f542f48e8824e581fd0f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:33.403461 2002 consensus_queue.cc:1035] T 012c8298a5d741909d6838470e8c05ba P 957a8a8be3f542f48e8824e581fd0f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 269.591 ms
time per row: 0.134796 ms
I20250627 13:52:33.729610 456 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49852:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250627 13:52:33.739856 1102 tablet_service.cc:1468] Processing CreateTablet for tablet a9953cb4c4af4d08b55f8f54f474438f (DEFAULT_TABLE table=rebalance_test_table_8 [id=16f401b8383b463d90c63e194c382025]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:33.740310 1102 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9953cb4c4af4d08b55f8f54f474438f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:33.740567 709 tablet_service.cc:1468] Processing CreateTablet for tablet 7bfa679d07d543dd8e678f2314fd328c (DEFAULT_TABLE table=rebalance_test_table_8 [id=16f401b8383b463d90c63e194c382025]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:33.740981 709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bfa679d07d543dd8e678f2314fd328c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:33.741494 840 tablet_service.cc:1468] Processing CreateTablet for tablet a9953cb4c4af4d08b55f8f54f474438f (DEFAULT_TABLE table=rebalance_test_table_8 [id=16f401b8383b463d90c63e194c382025]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:33.741513 972 tablet_service.cc:1468] Processing CreateTablet for tablet 7bfa679d07d543dd8e678f2314fd328c (DEFAULT_TABLE table=rebalance_test_table_8 [id=16f401b8383b463d90c63e194c382025]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:33.741886 840 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9953cb4c4af4d08b55f8f54f474438f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:33.741922 972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bfa679d07d543dd8e678f2314fd328c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:33.742425 841 tablet_service.cc:1468] Processing CreateTablet for tablet f296de1facde47308828eb4ed2283079 (DEFAULT_TABLE table=rebalance_test_table_8 [id=16f401b8383b463d90c63e194c382025]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:33.742753 841 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f296de1facde47308828eb4ed2283079. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:33.746064 578 tablet_service.cc:1468] Processing CreateTablet for tablet a9953cb4c4af4d08b55f8f54f474438f (DEFAULT_TABLE table=rebalance_test_table_8 [id=16f401b8383b463d90c63e194c382025]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:33.746105 1234 tablet_service.cc:1468] Processing CreateTablet for tablet 7bfa679d07d543dd8e678f2314fd328c (DEFAULT_TABLE table=rebalance_test_table_8 [id=16f401b8383b463d90c63e194c382025]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:33.746482 578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9953cb4c4af4d08b55f8f54f474438f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:33.746498 1234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bfa679d07d543dd8e678f2314fd328c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:33.750197 1103 tablet_service.cc:1468] Processing CreateTablet for tablet f296de1facde47308828eb4ed2283079 (DEFAULT_TABLE table=rebalance_test_table_8 [id=16f401b8383b463d90c63e194c382025]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:33.750674 1103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f296de1facde47308828eb4ed2283079. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:33.755530 577 tablet_service.cc:1468] Processing CreateTablet for tablet f296de1facde47308828eb4ed2283079 (DEFAULT_TABLE table=rebalance_test_table_8 [id=16f401b8383b463d90c63e194c382025]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:33.756048 577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f296de1facde47308828eb4ed2283079. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:33.758540 2077 tablet_bootstrap.cc:492] T 7bfa679d07d543dd8e678f2314fd328c P 50e8c97891d74b38b5c130d3bcbabcb6: Bootstrap starting.
I20250627 13:52:33.759686 2078 tablet_bootstrap.cc:492] T a9953cb4c4af4d08b55f8f54f474438f P fa66786862da49ae826fb519c7d89b5e: Bootstrap starting.
I20250627 13:52:33.762271 2077 tablet_bootstrap.cc:654] T 7bfa679d07d543dd8e678f2314fd328c P 50e8c97891d74b38b5c130d3bcbabcb6: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:33.762347 2078 tablet_bootstrap.cc:654] T a9953cb4c4af4d08b55f8f54f474438f P fa66786862da49ae826fb519c7d89b5e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:33.770690 2078 tablet_bootstrap.cc:492] T a9953cb4c4af4d08b55f8f54f474438f P fa66786862da49ae826fb519c7d89b5e: No bootstrap required, opened a new log
I20250627 13:52:33.771000 2078 ts_tablet_manager.cc:1397] T a9953cb4c4af4d08b55f8f54f474438f P fa66786862da49ae826fb519c7d89b5e: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.004s
I20250627 13:52:33.771963 2078 raft_consensus.cc:357] T a9953cb4c4af4d08b55f8f54f474438f P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:33.772285 2078 raft_consensus.cc:383] T a9953cb4c4af4d08b55f8f54f474438f P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:33.772433 2078 raft_consensus.cc:738] T a9953cb4c4af4d08b55f8f54f474438f P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa66786862da49ae826fb519c7d89b5e, State: Initialized, Role: FOLLOWER
I20250627 13:52:33.772718 2078 consensus_queue.cc:260] T a9953cb4c4af4d08b55f8f54f474438f P fa66786862da49ae826fb519c7d89b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:33.774117 2080 tablet_bootstrap.cc:492] T a9953cb4c4af4d08b55f8f54f474438f P 957a8a8be3f542f48e8824e581fd0f89: Bootstrap starting.
I20250627 13:52:33.774255 2078 ts_tablet_manager.cc:1428] T a9953cb4c4af4d08b55f8f54f474438f P fa66786862da49ae826fb519c7d89b5e: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250627 13:52:33.776234 2080 tablet_bootstrap.cc:654] T a9953cb4c4af4d08b55f8f54f474438f P 957a8a8be3f542f48e8824e581fd0f89: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:33.780238 2078 tablet_bootstrap.cc:492] T f296de1facde47308828eb4ed2283079 P fa66786862da49ae826fb519c7d89b5e: Bootstrap starting.
I20250627 13:52:33.781924 2078 tablet_bootstrap.cc:654] T f296de1facde47308828eb4ed2283079 P fa66786862da49ae826fb519c7d89b5e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:33.799072 2084 tablet_bootstrap.cc:492] T a9953cb4c4af4d08b55f8f54f474438f P 9f767210a91f44ae8b83b994131eee98: Bootstrap starting.
I20250627 13:52:33.802284 2077 tablet_bootstrap.cc:492] T 7bfa679d07d543dd8e678f2314fd328c P 50e8c97891d74b38b5c130d3bcbabcb6: No bootstrap required, opened a new log
I20250627 13:52:33.802441 2077 ts_tablet_manager.cc:1397] T 7bfa679d07d543dd8e678f2314fd328c P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent bootstrapping tablet: real 0.044s user 0.006s sys 0.003s
I20250627 13:52:33.803254 2077 raft_consensus.cc:357] T 7bfa679d07d543dd8e678f2314fd328c P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:33.803462 2077 raft_consensus.cc:383] T 7bfa679d07d543dd8e678f2314fd328c P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:33.803555 2077 raft_consensus.cc:738] T 7bfa679d07d543dd8e678f2314fd328c P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e8c97891d74b38b5c130d3bcbabcb6, State: Initialized, Role: FOLLOWER
I20250627 13:52:33.803761 2077 consensus_queue.cc:260] T 7bfa679d07d543dd8e678f2314fd328c P 50e8c97891d74b38b5c130d3bcbabcb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:33.805971 1790 raft_consensus.cc:491] T a9953cb4c4af4d08b55f8f54f474438f P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:33.806147 1790 raft_consensus.cc:513] T a9953cb4c4af4d08b55f8f54f474438f P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:33.806793 1790 leader_election.cc:290] T a9953cb4c4af4d08b55f8f54f474438f P fa66786862da49ae826fb519c7d89b5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69:42325), 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023)
I20250627 13:52:33.808221 1123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9953cb4c4af4d08b55f8f54f474438f" candidate_uuid: "fa66786862da49ae826fb519c7d89b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957a8a8be3f542f48e8824e581fd0f89" is_pre_election: true
W20250627 13:52:33.809724 797 leader_election.cc:343] T a9953cb4c4af4d08b55f8f54f474438f P fa66786862da49ae826fb519c7d89b5e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69:42325): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:52:33.809835 2086 tablet_bootstrap.cc:492] T 7bfa679d07d543dd8e678f2314fd328c P 75c4f59060974a3b823305b9dd339a7e: Bootstrap starting.
I20250627 13:52:33.812243 2086 tablet_bootstrap.cc:654] T 7bfa679d07d543dd8e678f2314fd328c P 75c4f59060974a3b823305b9dd339a7e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:33.813591 2079 tablet_bootstrap.cc:492] T 7bfa679d07d543dd8e678f2314fd328c P 5c28ff2fe9424379819d478fdd093cfc: Bootstrap starting.
I20250627 13:52:33.817333 2079 tablet_bootstrap.cc:654] T 7bfa679d07d543dd8e678f2314fd328c P 5c28ff2fe9424379819d478fdd093cfc: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:33.809303 598 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9953cb4c4af4d08b55f8f54f474438f" candidate_uuid: "fa66786862da49ae826fb519c7d89b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98" is_pre_election: true
W20250627 13:52:33.820020 794 leader_election.cc:343] T a9953cb4c4af4d08b55f8f54f474438f P fa66786862da49ae826fb519c7d89b5e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:52:33.820207 794 leader_election.cc:304] T a9953cb4c4af4d08b55f8f54f474438f P fa66786862da49ae826fb519c7d89b5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa66786862da49ae826fb519c7d89b5e; no voters: 957a8a8be3f542f48e8824e581fd0f89, 9f767210a91f44ae8b83b994131eee98
I20250627 13:52:33.820755 1790 raft_consensus.cc:2747] T a9953cb4c4af4d08b55f8f54f474438f P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250627 13:52:33.809280 2084 tablet_bootstrap.cc:654] T a9953cb4c4af4d08b55f8f54f474438f P 9f767210a91f44ae8b83b994131eee98: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:33.830277 2079 tablet_bootstrap.cc:492] T 7bfa679d07d543dd8e678f2314fd328c P 5c28ff2fe9424379819d478fdd093cfc: No bootstrap required, opened a new log
I20250627 13:52:33.830473 2079 ts_tablet_manager.cc:1397] T 7bfa679d07d543dd8e678f2314fd328c P 5c28ff2fe9424379819d478fdd093cfc: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.004s
I20250627 13:52:33.831333 2079 raft_consensus.cc:357] T 7bfa679d07d543dd8e678f2314fd328c P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:33.831553 2079 raft_consensus.cc:383] T 7bfa679d07d543dd8e678f2314fd328c P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:33.831678 2079 raft_consensus.cc:738] T 7bfa679d07d543dd8e678f2314fd328c P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c28ff2fe9424379819d478fdd093cfc, State: Initialized, Role: FOLLOWER
I20250627 13:52:33.831986 2079 consensus_queue.cc:260] T 7bfa679d07d543dd8e678f2314fd328c P 5c28ff2fe9424379819d478fdd093cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:33.833572 2079 ts_tablet_manager.cc:1428] T 7bfa679d07d543dd8e678f2314fd328c P 5c28ff2fe9424379819d478fdd093cfc: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250627 13:52:33.838762 2086 tablet_bootstrap.cc:492] T 7bfa679d07d543dd8e678f2314fd328c P 75c4f59060974a3b823305b9dd339a7e: No bootstrap required, opened a new log
I20250627 13:52:33.839334 2077 ts_tablet_manager.cc:1428] T 7bfa679d07d543dd8e678f2314fd328c P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent starting tablet: real 0.037s user 0.011s sys 0.006s
I20250627 13:52:33.840576 2086 ts_tablet_manager.cc:1397] T 7bfa679d07d543dd8e678f2314fd328c P 75c4f59060974a3b823305b9dd339a7e: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.003s
I20250627 13:52:33.842051 2078 tablet_bootstrap.cc:492] T f296de1facde47308828eb4ed2283079 P fa66786862da49ae826fb519c7d89b5e: No bootstrap required, opened a new log
I20250627 13:52:33.842423 2078 ts_tablet_manager.cc:1397] T f296de1facde47308828eb4ed2283079 P fa66786862da49ae826fb519c7d89b5e: Time spent bootstrapping tablet: real 0.063s user 0.014s sys 0.019s
I20250627 13:52:33.843231 2080 tablet_bootstrap.cc:492] T a9953cb4c4af4d08b55f8f54f474438f P 957a8a8be3f542f48e8824e581fd0f89: No bootstrap required, opened a new log
I20250627 13:52:33.843181 2078 raft_consensus.cc:357] T f296de1facde47308828eb4ed2283079 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:33.843361 2078 raft_consensus.cc:383] T f296de1facde47308828eb4ed2283079 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:33.843389 2080 ts_tablet_manager.cc:1397] T a9953cb4c4af4d08b55f8f54f474438f P 957a8a8be3f542f48e8824e581fd0f89: Time spent bootstrapping tablet: real 0.069s user 0.024s sys 0.040s
I20250627 13:52:33.843425 2078 raft_consensus.cc:738] T f296de1facde47308828eb4ed2283079 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa66786862da49ae826fb519c7d89b5e, State: Initialized, Role: FOLLOWER
I20250627 13:52:33.843686 2078 consensus_queue.cc:260] T f296de1facde47308828eb4ed2283079 P fa66786862da49ae826fb519c7d89b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:33.844229 2078 ts_tablet_manager.cc:1428] T f296de1facde47308828eb4ed2283079 P fa66786862da49ae826fb519c7d89b5e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250627 13:52:33.844220 2080 raft_consensus.cc:357] T a9953cb4c4af4d08b55f8f54f474438f P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:33.844396 2080 raft_consensus.cc:383] T a9953cb4c4af4d08b55f8f54f474438f P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:33.844475 2080 raft_consensus.cc:738] T a9953cb4c4af4d08b55f8f54f474438f P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957a8a8be3f542f48e8824e581fd0f89, State: Initialized, Role: FOLLOWER
I20250627 13:52:33.844676 2080 consensus_queue.cc:260] T a9953cb4c4af4d08b55f8f54f474438f P 957a8a8be3f542f48e8824e581fd0f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:33.853998 2080 ts_tablet_manager.cc:1428] T a9953cb4c4af4d08b55f8f54f474438f P 957a8a8be3f542f48e8824e581fd0f89: Time spent starting tablet: real 0.011s user 0.001s sys 0.002s
I20250627 13:52:33.854338 2080 tablet_bootstrap.cc:492] T f296de1facde47308828eb4ed2283079 P 957a8a8be3f542f48e8824e581fd0f89: Bootstrap starting.
I20250627 13:52:33.856514 2080 tablet_bootstrap.cc:654] T f296de1facde47308828eb4ed2283079 P 957a8a8be3f542f48e8824e581fd0f89: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:33.861825 2086 raft_consensus.cc:357] T 7bfa679d07d543dd8e678f2314fd328c P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:33.867811 2086 raft_consensus.cc:383] T 7bfa679d07d543dd8e678f2314fd328c P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:33.868042 2086 raft_consensus.cc:738] T 7bfa679d07d543dd8e678f2314fd328c P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c4f59060974a3b823305b9dd339a7e, State: Initialized, Role: FOLLOWER
I20250627 13:52:33.868484 2086 consensus_queue.cc:260] T 7bfa679d07d543dd8e678f2314fd328c P 75c4f59060974a3b823305b9dd339a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:33.871145 2084 tablet_bootstrap.cc:492] T a9953cb4c4af4d08b55f8f54f474438f P 9f767210a91f44ae8b83b994131eee98: No bootstrap required, opened a new log
I20250627 13:52:33.871388 2084 ts_tablet_manager.cc:1397] T a9953cb4c4af4d08b55f8f54f474438f P 9f767210a91f44ae8b83b994131eee98: Time spent bootstrapping tablet: real 0.072s user 0.006s sys 0.000s
I20250627 13:52:33.872174 2084 raft_consensus.cc:357] T a9953cb4c4af4d08b55f8f54f474438f P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:33.872407 2080 tablet_bootstrap.cc:492] T f296de1facde47308828eb4ed2283079 P 957a8a8be3f542f48e8824e581fd0f89: No bootstrap required, opened a new log
I20250627 13:52:33.872431 2084 raft_consensus.cc:383] T a9953cb4c4af4d08b55f8f54f474438f P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:33.872583 2080 ts_tablet_manager.cc:1397] T f296de1facde47308828eb4ed2283079 P 957a8a8be3f542f48e8824e581fd0f89: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.007s
I20250627 13:52:33.872944 2084 raft_consensus.cc:738] T a9953cb4c4af4d08b55f8f54f474438f P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f767210a91f44ae8b83b994131eee98, State: Initialized, Role: FOLLOWER
I20250627 13:52:33.873251 2084 consensus_queue.cc:260] T a9953cb4c4af4d08b55f8f54f474438f P 9f767210a91f44ae8b83b994131eee98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:33.873359 2080 raft_consensus.cc:357] T f296de1facde47308828eb4ed2283079 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:33.873557 2080 raft_consensus.cc:383] T f296de1facde47308828eb4ed2283079 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:33.873636 2080 raft_consensus.cc:738] T f296de1facde47308828eb4ed2283079 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957a8a8be3f542f48e8824e581fd0f89, State: Initialized, Role: FOLLOWER
I20250627 13:52:33.873940 2080 consensus_queue.cc:260] T f296de1facde47308828eb4ed2283079 P 957a8a8be3f542f48e8824e581fd0f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:33.874420 2080 ts_tablet_manager.cc:1428] T f296de1facde47308828eb4ed2283079 P 957a8a8be3f542f48e8824e581fd0f89: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250627 13:52:33.875170 2084 ts_tablet_manager.cc:1428] T a9953cb4c4af4d08b55f8f54f474438f P 9f767210a91f44ae8b83b994131eee98: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250627 13:52:33.875418 2084 tablet_bootstrap.cc:492] T f296de1facde47308828eb4ed2283079 P 9f767210a91f44ae8b83b994131eee98: Bootstrap starting.
I20250627 13:52:33.877082 2086 ts_tablet_manager.cc:1428] T 7bfa679d07d543dd8e678f2314fd328c P 75c4f59060974a3b823305b9dd339a7e: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20250627 13:52:33.878418 2084 tablet_bootstrap.cc:654] T f296de1facde47308828eb4ed2283079 P 9f767210a91f44ae8b83b994131eee98: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:33.879112 1954 raft_consensus.cc:491] T a9953cb4c4af4d08b55f8f54f474438f P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:33.879279 1954 raft_consensus.cc:513] T a9953cb4c4af4d08b55f8f54f474438f P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:33.880123 1954 leader_election.cc:290] T a9953cb4c4af4d08b55f8f54f474438f P 9f767210a91f44ae8b83b994131eee98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747), 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69:42325)
I20250627 13:52:33.880406 861 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9953cb4c4af4d08b55f8f54f474438f" candidate_uuid: "9f767210a91f44ae8b83b994131eee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e" is_pre_election: true
I20250627 13:52:33.880635 861 raft_consensus.cc:2466] T a9953cb4c4af4d08b55f8f54f474438f P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f767210a91f44ae8b83b994131eee98 in term 0.
I20250627 13:52:33.880831 1123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9953cb4c4af4d08b55f8f54f474438f" candidate_uuid: "9f767210a91f44ae8b83b994131eee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957a8a8be3f542f48e8824e581fd0f89" is_pre_election: true
I20250627 13:52:33.881043 1123 raft_consensus.cc:2466] T a9953cb4c4af4d08b55f8f54f474438f P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f767210a91f44ae8b83b994131eee98 in term 0.
I20250627 13:52:33.881095 533 leader_election.cc:304] T a9953cb4c4af4d08b55f8f54f474438f P 9f767210a91f44ae8b83b994131eee98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f767210a91f44ae8b83b994131eee98, fa66786862da49ae826fb519c7d89b5e; no voters:
I20250627 13:52:33.883567 1954 raft_consensus.cc:2802] T a9953cb4c4af4d08b55f8f54f474438f P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:33.883718 1954 raft_consensus.cc:491] T a9953cb4c4af4d08b55f8f54f474438f P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:33.883821 1954 raft_consensus.cc:3058] T a9953cb4c4af4d08b55f8f54f474438f P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:33.886962 2084 tablet_bootstrap.cc:492] T f296de1facde47308828eb4ed2283079 P 9f767210a91f44ae8b83b994131eee98: No bootstrap required, opened a new log
I20250627 13:52:33.887110 2084 ts_tablet_manager.cc:1397] T f296de1facde47308828eb4ed2283079 P 9f767210a91f44ae8b83b994131eee98: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250627 13:52:33.887704 2084 raft_consensus.cc:357] T f296de1facde47308828eb4ed2283079 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:33.887852 1954 raft_consensus.cc:513] T a9953cb4c4af4d08b55f8f54f474438f P 9f767210a91f44ae8b83b994131eee98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:33.888954 861 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9953cb4c4af4d08b55f8f54f474438f" candidate_uuid: "9f767210a91f44ae8b83b994131eee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e"
I20250627 13:52:33.889223 861 raft_consensus.cc:3058] T a9953cb4c4af4d08b55f8f54f474438f P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:33.889773 2084 raft_consensus.cc:383] T f296de1facde47308828eb4ed2283079 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:33.889919 2084 raft_consensus.cc:738] T f296de1facde47308828eb4ed2283079 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f767210a91f44ae8b83b994131eee98, State: Initialized, Role: FOLLOWER
I20250627 13:52:33.890182 2084 consensus_queue.cc:260] T f296de1facde47308828eb4ed2283079 P 9f767210a91f44ae8b83b994131eee98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:33.890971 1954 leader_election.cc:290] T a9953cb4c4af4d08b55f8f54f474438f P 9f767210a91f44ae8b83b994131eee98 [CANDIDATE]: Term 1 election: Requested vote from peers fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747), 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69:42325)
I20250627 13:52:33.892710 1123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a9953cb4c4af4d08b55f8f54f474438f" candidate_uuid: "9f767210a91f44ae8b83b994131eee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957a8a8be3f542f48e8824e581fd0f89"
I20250627 13:52:33.892918 1123 raft_consensus.cc:3058] T a9953cb4c4af4d08b55f8f54f474438f P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:33.896273 1123 raft_consensus.cc:2466] T a9953cb4c4af4d08b55f8f54f474438f P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f767210a91f44ae8b83b994131eee98 in term 1.
I20250627 13:52:33.896772 534 leader_election.cc:304] T a9953cb4c4af4d08b55f8f54f474438f P 9f767210a91f44ae8b83b994131eee98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, 9f767210a91f44ae8b83b994131eee98; no voters:
I20250627 13:52:33.899541 861 raft_consensus.cc:2466] T a9953cb4c4af4d08b55f8f54f474438f P fa66786862da49ae826fb519c7d89b5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f767210a91f44ae8b83b994131eee98 in term 1.
I20250627 13:52:33.900290 1954 raft_consensus.cc:2802] T a9953cb4c4af4d08b55f8f54f474438f P 9f767210a91f44ae8b83b994131eee98 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:33.900449 1954 raft_consensus.cc:695] T a9953cb4c4af4d08b55f8f54f474438f P 9f767210a91f44ae8b83b994131eee98 [term 1 LEADER]: Becoming Leader. State: Replica: 9f767210a91f44ae8b83b994131eee98, State: Running, Role: LEADER
I20250627 13:52:33.900784 1954 consensus_queue.cc:237] T a9953cb4c4af4d08b55f8f54f474438f P 9f767210a91f44ae8b83b994131eee98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:33.902822 2084 ts_tablet_manager.cc:1428] T f296de1facde47308828eb4ed2283079 P 9f767210a91f44ae8b83b994131eee98: Time spent starting tablet: real 0.016s user 0.005s sys 0.008s
I20250627 13:52:33.904556 456 catalog_manager.cc:5582] T a9953cb4c4af4d08b55f8f54f474438f P 9f767210a91f44ae8b83b994131eee98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f767210a91f44ae8b83b994131eee98 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "9f767210a91f44ae8b83b994131eee98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } health_report { overall_health: HEALTHY } } }
I20250627 13:52:33.937201 2002 raft_consensus.cc:491] T f296de1facde47308828eb4ed2283079 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:33.937407 2002 raft_consensus.cc:513] T f296de1facde47308828eb4ed2283079 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:33.938004 2002 leader_election.cc:290] T f296de1facde47308828eb4ed2283079 P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747), 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023)
I20250627 13:52:33.938486 861 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f296de1facde47308828eb4ed2283079" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e" is_pre_election: true
I20250627 13:52:33.938493 598 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f296de1facde47308828eb4ed2283079" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98" is_pre_election: true
I20250627 13:52:33.938741 598 raft_consensus.cc:2466] T f296de1facde47308828eb4ed2283079 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 0.
I20250627 13:52:33.938758 861 raft_consensus.cc:2466] T f296de1facde47308828eb4ed2283079 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 0.
I20250627 13:52:33.939157 1058 leader_election.cc:304] T f296de1facde47308828eb4ed2283079 P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, fa66786862da49ae826fb519c7d89b5e; no voters:
I20250627 13:52:33.939462 2002 raft_consensus.cc:2802] T f296de1facde47308828eb4ed2283079 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:33.939580 2002 raft_consensus.cc:491] T f296de1facde47308828eb4ed2283079 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:33.939715 2002 raft_consensus.cc:3058] T f296de1facde47308828eb4ed2283079 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:33.943188 2002 raft_consensus.cc:513] T f296de1facde47308828eb4ed2283079 P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:33.943981 2002 leader_election.cc:290] T f296de1facde47308828eb4ed2283079 P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 election: Requested vote from peers fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747), 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023)
I20250627 13:52:33.944377 598 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f296de1facde47308828eb4ed2283079" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98"
I20250627 13:52:33.944365 861 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f296de1facde47308828eb4ed2283079" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e"
I20250627 13:52:33.944587 598 raft_consensus.cc:3058] T f296de1facde47308828eb4ed2283079 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:33.944597 861 raft_consensus.cc:3058] T f296de1facde47308828eb4ed2283079 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:33.946599 598 raft_consensus.cc:2466] T f296de1facde47308828eb4ed2283079 P 9f767210a91f44ae8b83b994131eee98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 1.
I20250627 13:52:33.947009 1056 leader_election.cc:304] T f296de1facde47308828eb4ed2283079 P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, 9f767210a91f44ae8b83b994131eee98; no voters:
I20250627 13:52:33.947281 2002 raft_consensus.cc:2802] T f296de1facde47308828eb4ed2283079 P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:33.947458 2002 raft_consensus.cc:695] T f296de1facde47308828eb4ed2283079 P 957a8a8be3f542f48e8824e581fd0f89 [term 1 LEADER]: Becoming Leader. State: Replica: 957a8a8be3f542f48e8824e581fd0f89, State: Running, Role: LEADER
I20250627 13:52:33.947763 2002 consensus_queue.cc:237] T f296de1facde47308828eb4ed2283079 P 957a8a8be3f542f48e8824e581fd0f89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:33.947925 861 raft_consensus.cc:2466] T f296de1facde47308828eb4ed2283079 P fa66786862da49ae826fb519c7d89b5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 1.
I20250627 13:52:33.951346 456 catalog_manager.cc:5582] T f296de1facde47308828eb4ed2283079 P 957a8a8be3f542f48e8824e581fd0f89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69). New cstate: current_term: 1 leader_uuid: "957a8a8be3f542f48e8824e581fd0f89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } health_report { overall_health: HEALTHY } } }
I20250627 13:52:34.160470 2087 raft_consensus.cc:491] T 7bfa679d07d543dd8e678f2314fd328c P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:34.160717 2087 raft_consensus.cc:513] T 7bfa679d07d543dd8e678f2314fd328c P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:34.161484 2087 leader_election.cc:290] T 7bfa679d07d543dd8e678f2314fd328c P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499), 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70:39887)
I20250627 13:52:34.162022 728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7bfa679d07d543dd8e678f2314fd328c" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c4f59060974a3b823305b9dd339a7e" is_pre_election: true
I20250627 13:52:34.162211 728 raft_consensus.cc:2466] T 7bfa679d07d543dd8e678f2314fd328c P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 0.
I20250627 13:52:34.162231 1253 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7bfa679d07d543dd8e678f2314fd328c" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c28ff2fe9424379819d478fdd093cfc" is_pre_election: true
I20250627 13:52:34.162457 1253 raft_consensus.cc:2466] T 7bfa679d07d543dd8e678f2314fd328c P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 0.
I20250627 13:52:34.162631 928 leader_election.cc:304] T 7bfa679d07d543dd8e678f2314fd328c P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50e8c97891d74b38b5c130d3bcbabcb6, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:34.162909 2087 raft_consensus.cc:2802] T 7bfa679d07d543dd8e678f2314fd328c P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:34.163019 2087 raft_consensus.cc:491] T 7bfa679d07d543dd8e678f2314fd328c P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:34.163100 2087 raft_consensus.cc:3058] T 7bfa679d07d543dd8e678f2314fd328c P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:34.166322 2087 raft_consensus.cc:513] T 7bfa679d07d543dd8e678f2314fd328c P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:34.166886 2087 leader_election.cc:290] T 7bfa679d07d543dd8e678f2314fd328c P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 election: Requested vote from peers 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499), 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70:39887)
I20250627 13:52:34.167266 728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7bfa679d07d543dd8e678f2314fd328c" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c4f59060974a3b823305b9dd339a7e"
I20250627 13:52:34.167328 1253 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7bfa679d07d543dd8e678f2314fd328c" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c28ff2fe9424379819d478fdd093cfc"
I20250627 13:52:34.167460 728 raft_consensus.cc:3058] T 7bfa679d07d543dd8e678f2314fd328c P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:34.167528 1253 raft_consensus.cc:3058] T 7bfa679d07d543dd8e678f2314fd328c P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:34.169240 728 raft_consensus.cc:2466] T 7bfa679d07d543dd8e678f2314fd328c P 75c4f59060974a3b823305b9dd339a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 1.
I20250627 13:52:34.169307 1253 raft_consensus.cc:2466] T 7bfa679d07d543dd8e678f2314fd328c P 5c28ff2fe9424379819d478fdd093cfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 1.
I20250627 13:52:34.169636 928 leader_election.cc:304] T 7bfa679d07d543dd8e678f2314fd328c P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50e8c97891d74b38b5c130d3bcbabcb6, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:34.170076 2087 raft_consensus.cc:2802] T 7bfa679d07d543dd8e678f2314fd328c P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:34.170215 2087 raft_consensus.cc:695] T 7bfa679d07d543dd8e678f2314fd328c P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 LEADER]: Becoming Leader. State: Replica: 50e8c97891d74b38b5c130d3bcbabcb6, State: Running, Role: LEADER
I20250627 13:52:34.170480 2087 consensus_queue.cc:237] T 7bfa679d07d543dd8e678f2314fd328c P 50e8c97891d74b38b5c130d3bcbabcb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:34.173393 456 catalog_manager.cc:5582] T 7bfa679d07d543dd8e678f2314fd328c P 50e8c97891d74b38b5c130d3bcbabcb6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68). New cstate: current_term: 1 leader_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } health_report { overall_health: UNKNOWN } } }
I20250627 13:52:34.337230 2002 consensus_queue.cc:1035] T f296de1facde47308828eb4ed2283079 P 957a8a8be3f542f48e8824e581fd0f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:34.358294 2002 consensus_queue.cc:1035] T f296de1facde47308828eb4ed2283079 P 957a8a8be3f542f48e8824e581fd0f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:34.398376 2093 consensus_queue.cc:1035] T a9953cb4c4af4d08b55f8f54f474438f P 9f767210a91f44ae8b83b994131eee98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:34.402899 2093 consensus_queue.cc:1035] T a9953cb4c4af4d08b55f8f54f474438f P 9f767210a91f44ae8b83b994131eee98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250627 13:52:34.575486 2097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:34.575852 2097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:34.583956 2097 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250627 13:52:34.604135 2087 consensus_queue.cc:1035] T 7bfa679d07d543dd8e678f2314fd328c P 50e8c97891d74b38b5c130d3bcbabcb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:34.612679 2087 consensus_queue.cc:1035] T 7bfa679d07d543dd8e678f2314fd328c P 50e8c97891d74b38b5c130d3bcbabcb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
INSERT report
rows total: 2000
time total: 284.182 ms
time per row: 0.142091 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20250627 13:52:36.032652 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49852:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250627 13:52:36.041530 1234 tablet_service.cc:1468] Processing CreateTablet for tablet b9b2432425534c4db188ac324295513d (DEFAULT_TABLE table=rebalance_test_table_9 [id=b4cf33caffb34fd097251286b505ea9a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:36.042019 1234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9b2432425534c4db188ac324295513d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:36.042171 972 tablet_service.cc:1468] Processing CreateTablet for tablet b9b2432425534c4db188ac324295513d (DEFAULT_TABLE table=rebalance_test_table_9 [id=b4cf33caffb34fd097251286b505ea9a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:36.042564 972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9b2432425534c4db188ac324295513d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:36.042706 1233 tablet_service.cc:1468] Processing CreateTablet for tablet 32895feb7def4132953f5a4d6bf0bbd8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=b4cf33caffb34fd097251286b505ea9a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:36.043063 1233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32895feb7def4132953f5a4d6bf0bbd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:36.043761 578 tablet_service.cc:1468] Processing CreateTablet for tablet 574d86438dc240aca8fd3791c70d19b3 (DEFAULT_TABLE table=rebalance_test_table_9 [id=b4cf33caffb34fd097251286b505ea9a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:36.044157 578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 574d86438dc240aca8fd3791c70d19b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:36.046005 709 tablet_service.cc:1468] Processing CreateTablet for tablet b9b2432425534c4db188ac324295513d (DEFAULT_TABLE table=rebalance_test_table_9 [id=b4cf33caffb34fd097251286b505ea9a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:36.046361 709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9b2432425534c4db188ac324295513d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:36.047322 710 tablet_service.cc:1468] Processing CreateTablet for tablet 32895feb7def4132953f5a4d6bf0bbd8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=b4cf33caffb34fd097251286b505ea9a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:36.047719 710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32895feb7def4132953f5a4d6bf0bbd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:36.048024 971 tablet_service.cc:1468] Processing CreateTablet for tablet 32895feb7def4132953f5a4d6bf0bbd8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=b4cf33caffb34fd097251286b505ea9a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:36.048404 971 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32895feb7def4132953f5a4d6bf0bbd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:36.053961 841 tablet_service.cc:1468] Processing CreateTablet for tablet 574d86438dc240aca8fd3791c70d19b3 (DEFAULT_TABLE table=rebalance_test_table_9 [id=b4cf33caffb34fd097251286b505ea9a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:36.054463 841 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 574d86438dc240aca8fd3791c70d19b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:36.055837 2170 tablet_bootstrap.cc:492] T b9b2432425534c4db188ac324295513d P 75c4f59060974a3b823305b9dd339a7e: Bootstrap starting.
I20250627 13:52:36.056100 2169 tablet_bootstrap.cc:492] T b9b2432425534c4db188ac324295513d P 50e8c97891d74b38b5c130d3bcbabcb6: Bootstrap starting.
I20250627 13:52:36.057605 2170 tablet_bootstrap.cc:654] T b9b2432425534c4db188ac324295513d P 75c4f59060974a3b823305b9dd339a7e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:36.057885 2169 tablet_bootstrap.cc:654] T b9b2432425534c4db188ac324295513d P 50e8c97891d74b38b5c130d3bcbabcb6: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:36.072819 2173 tablet_bootstrap.cc:492] T b9b2432425534c4db188ac324295513d P 5c28ff2fe9424379819d478fdd093cfc: Bootstrap starting.
I20250627 13:52:36.078287 2175 tablet_bootstrap.cc:492] T 574d86438dc240aca8fd3791c70d19b3 P 9f767210a91f44ae8b83b994131eee98: Bootstrap starting.
I20250627 13:52:36.078867 2173 tablet_bootstrap.cc:654] T b9b2432425534c4db188ac324295513d P 5c28ff2fe9424379819d478fdd093cfc: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:36.080092 2175 tablet_bootstrap.cc:654] T 574d86438dc240aca8fd3791c70d19b3 P 9f767210a91f44ae8b83b994131eee98: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:36.081254 2169 tablet_bootstrap.cc:492] T b9b2432425534c4db188ac324295513d P 50e8c97891d74b38b5c130d3bcbabcb6: No bootstrap required, opened a new log
I20250627 13:52:36.081494 2169 ts_tablet_manager.cc:1397] T b9b2432425534c4db188ac324295513d P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.021s
I20250627 13:52:36.082412 2169 raft_consensus.cc:357] T b9b2432425534c4db188ac324295513d P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:36.082718 2169 raft_consensus.cc:383] T b9b2432425534c4db188ac324295513d P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:36.082872 2169 raft_consensus.cc:738] T b9b2432425534c4db188ac324295513d P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e8c97891d74b38b5c130d3bcbabcb6, State: Initialized, Role: FOLLOWER
I20250627 13:52:36.083189 2169 consensus_queue.cc:260] T b9b2432425534c4db188ac324295513d P 50e8c97891d74b38b5c130d3bcbabcb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:36.083842 2169 ts_tablet_manager.cc:1428] T b9b2432425534c4db188ac324295513d P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:52:36.084234 2169 tablet_bootstrap.cc:492] T 32895feb7def4132953f5a4d6bf0bbd8 P 50e8c97891d74b38b5c130d3bcbabcb6: Bootstrap starting.
I20250627 13:52:36.085981 2173 tablet_bootstrap.cc:492] T b9b2432425534c4db188ac324295513d P 5c28ff2fe9424379819d478fdd093cfc: No bootstrap required, opened a new log
I20250627 13:52:36.086071 2169 tablet_bootstrap.cc:654] T 32895feb7def4132953f5a4d6bf0bbd8 P 50e8c97891d74b38b5c130d3bcbabcb6: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:36.086181 2173 ts_tablet_manager.cc:1397] T b9b2432425534c4db188ac324295513d P 5c28ff2fe9424379819d478fdd093cfc: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.004s
I20250627 13:52:36.087056 2173 raft_consensus.cc:357] T b9b2432425534c4db188ac324295513d P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:36.087342 2173 raft_consensus.cc:383] T b9b2432425534c4db188ac324295513d P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:36.087407 2176 tablet_bootstrap.cc:492] T 574d86438dc240aca8fd3791c70d19b3 P fa66786862da49ae826fb519c7d89b5e: Bootstrap starting.
I20250627 13:52:36.087495 2173 raft_consensus.cc:738] T b9b2432425534c4db188ac324295513d P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c28ff2fe9424379819d478fdd093cfc, State: Initialized, Role: FOLLOWER
I20250627 13:52:36.087849 2173 consensus_queue.cc:260] T b9b2432425534c4db188ac324295513d P 5c28ff2fe9424379819d478fdd093cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:36.090220 2173 ts_tablet_manager.cc:1428] T b9b2432425534c4db188ac324295513d P 5c28ff2fe9424379819d478fdd093cfc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250627 13:52:36.091086 2169 tablet_bootstrap.cc:492] T 32895feb7def4132953f5a4d6bf0bbd8 P 50e8c97891d74b38b5c130d3bcbabcb6: No bootstrap required, opened a new log
I20250627 13:52:36.091308 2169 ts_tablet_manager.cc:1397] T 32895feb7def4132953f5a4d6bf0bbd8 P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20250627 13:52:36.091507 1103 tablet_service.cc:1468] Processing CreateTablet for tablet 574d86438dc240aca8fd3791c70d19b3 (DEFAULT_TABLE table=rebalance_test_table_9 [id=b4cf33caffb34fd097251286b505ea9a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:36.091691 2173 tablet_bootstrap.cc:492] T 32895feb7def4132953f5a4d6bf0bbd8 P 5c28ff2fe9424379819d478fdd093cfc: Bootstrap starting.
I20250627 13:52:36.091929 1103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 574d86438dc240aca8fd3791c70d19b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:36.092074 2169 raft_consensus.cc:357] T 32895feb7def4132953f5a4d6bf0bbd8 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:36.092275 2169 raft_consensus.cc:383] T 32895feb7def4132953f5a4d6bf0bbd8 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:36.092346 2169 raft_consensus.cc:738] T 32895feb7def4132953f5a4d6bf0bbd8 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e8c97891d74b38b5c130d3bcbabcb6, State: Initialized, Role: FOLLOWER
I20250627 13:52:36.092548 2175 tablet_bootstrap.cc:492] T 574d86438dc240aca8fd3791c70d19b3 P 9f767210a91f44ae8b83b994131eee98: No bootstrap required, opened a new log
I20250627 13:52:36.092675 2175 ts_tablet_manager.cc:1397] T 574d86438dc240aca8fd3791c70d19b3 P 9f767210a91f44ae8b83b994131eee98: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20250627 13:52:36.092594 2169 consensus_queue.cc:260] T 32895feb7def4132953f5a4d6bf0bbd8 P 50e8c97891d74b38b5c130d3bcbabcb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:36.093164 2169 ts_tablet_manager.cc:1428] T 32895feb7def4132953f5a4d6bf0bbd8 P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250627 13:52:36.093279 2173 tablet_bootstrap.cc:654] T 32895feb7def4132953f5a4d6bf0bbd8 P 5c28ff2fe9424379819d478fdd093cfc: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:36.094061 2175 raft_consensus.cc:357] T 574d86438dc240aca8fd3791c70d19b3 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:36.094293 2175 raft_consensus.cc:383] T 574d86438dc240aca8fd3791c70d19b3 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:36.094380 2175 raft_consensus.cc:738] T 574d86438dc240aca8fd3791c70d19b3 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f767210a91f44ae8b83b994131eee98, State: Initialized, Role: FOLLOWER
I20250627 13:52:36.094571 2175 consensus_queue.cc:260] T 574d86438dc240aca8fd3791c70d19b3 P 9f767210a91f44ae8b83b994131eee98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:36.095774 2175 ts_tablet_manager.cc:1428] T 574d86438dc240aca8fd3791c70d19b3 P 9f767210a91f44ae8b83b994131eee98: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250627 13:52:36.098704 2173 tablet_bootstrap.cc:492] T 32895feb7def4132953f5a4d6bf0bbd8 P 5c28ff2fe9424379819d478fdd093cfc: No bootstrap required, opened a new log
I20250627 13:52:36.098850 2173 ts_tablet_manager.cc:1397] T 32895feb7def4132953f5a4d6bf0bbd8 P 5c28ff2fe9424379819d478fdd093cfc: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20250627 13:52:36.099424 2173 raft_consensus.cc:357] T 32895feb7def4132953f5a4d6bf0bbd8 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:36.099611 2173 raft_consensus.cc:383] T 32895feb7def4132953f5a4d6bf0bbd8 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:36.099694 2173 raft_consensus.cc:738] T 32895feb7def4132953f5a4d6bf0bbd8 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c28ff2fe9424379819d478fdd093cfc, State: Initialized, Role: FOLLOWER
I20250627 13:52:36.099946 2173 consensus_queue.cc:260] T 32895feb7def4132953f5a4d6bf0bbd8 P 5c28ff2fe9424379819d478fdd093cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:36.101241 2173 ts_tablet_manager.cc:1428] T 32895feb7def4132953f5a4d6bf0bbd8 P 5c28ff2fe9424379819d478fdd093cfc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250627 13:52:36.103699 2176 tablet_bootstrap.cc:654] T 574d86438dc240aca8fd3791c70d19b3 P fa66786862da49ae826fb519c7d89b5e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:36.106267 2170 tablet_bootstrap.cc:492] T b9b2432425534c4db188ac324295513d P 75c4f59060974a3b823305b9dd339a7e: No bootstrap required, opened a new log
I20250627 13:52:36.106530 2170 ts_tablet_manager.cc:1397] T b9b2432425534c4db188ac324295513d P 75c4f59060974a3b823305b9dd339a7e: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.019s
I20250627 13:52:36.107574 2170 raft_consensus.cc:357] T b9b2432425534c4db188ac324295513d P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:36.107838 2170 raft_consensus.cc:383] T b9b2432425534c4db188ac324295513d P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:36.107951 2170 raft_consensus.cc:738] T b9b2432425534c4db188ac324295513d P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c4f59060974a3b823305b9dd339a7e, State: Initialized, Role: FOLLOWER
I20250627 13:52:36.108232 2170 consensus_queue.cc:260] T b9b2432425534c4db188ac324295513d P 75c4f59060974a3b823305b9dd339a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:36.108994 2170 ts_tablet_manager.cc:1428] T b9b2432425534c4db188ac324295513d P 75c4f59060974a3b823305b9dd339a7e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:52:36.109447 2170 tablet_bootstrap.cc:492] T 32895feb7def4132953f5a4d6bf0bbd8 P 75c4f59060974a3b823305b9dd339a7e: Bootstrap starting.
I20250627 13:52:36.110404 2182 tablet_bootstrap.cc:492] T 574d86438dc240aca8fd3791c70d19b3 P 957a8a8be3f542f48e8824e581fd0f89: Bootstrap starting.
I20250627 13:52:36.111927 2170 tablet_bootstrap.cc:654] T 32895feb7def4132953f5a4d6bf0bbd8 P 75c4f59060974a3b823305b9dd339a7e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:36.113541 2182 tablet_bootstrap.cc:654] T 574d86438dc240aca8fd3791c70d19b3 P 957a8a8be3f542f48e8824e581fd0f89: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:36.114598 2176 tablet_bootstrap.cc:492] T 574d86438dc240aca8fd3791c70d19b3 P fa66786862da49ae826fb519c7d89b5e: No bootstrap required, opened a new log
I20250627 13:52:36.114742 2176 ts_tablet_manager.cc:1397] T 574d86438dc240aca8fd3791c70d19b3 P fa66786862da49ae826fb519c7d89b5e: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.000s
I20250627 13:52:36.115396 2176 raft_consensus.cc:357] T 574d86438dc240aca8fd3791c70d19b3 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:36.115548 2176 raft_consensus.cc:383] T 574d86438dc240aca8fd3791c70d19b3 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:36.115654 2176 raft_consensus.cc:738] T 574d86438dc240aca8fd3791c70d19b3 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa66786862da49ae826fb519c7d89b5e, State: Initialized, Role: FOLLOWER
I20250627 13:52:36.115836 2176 consensus_queue.cc:260] T 574d86438dc240aca8fd3791c70d19b3 P fa66786862da49ae826fb519c7d89b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:36.116230 2176 ts_tablet_manager.cc:1428] T 574d86438dc240aca8fd3791c70d19b3 P fa66786862da49ae826fb519c7d89b5e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250627 13:52:36.117015 2182 tablet_bootstrap.cc:492] T 574d86438dc240aca8fd3791c70d19b3 P 957a8a8be3f542f48e8824e581fd0f89: No bootstrap required, opened a new log
I20250627 13:52:36.117159 2182 ts_tablet_manager.cc:1397] T 574d86438dc240aca8fd3791c70d19b3 P 957a8a8be3f542f48e8824e581fd0f89: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20250627 13:52:36.117970 2182 raft_consensus.cc:357] T 574d86438dc240aca8fd3791c70d19b3 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:36.118151 2182 raft_consensus.cc:383] T 574d86438dc240aca8fd3791c70d19b3 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:36.118252 2182 raft_consensus.cc:738] T 574d86438dc240aca8fd3791c70d19b3 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957a8a8be3f542f48e8824e581fd0f89, State: Initialized, Role: FOLLOWER
I20250627 13:52:36.118459 2182 consensus_queue.cc:260] T 574d86438dc240aca8fd3791c70d19b3 P 957a8a8be3f542f48e8824e581fd0f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:36.118968 2182 ts_tablet_manager.cc:1428] T 574d86438dc240aca8fd3791c70d19b3 P 957a8a8be3f542f48e8824e581fd0f89: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:52:36.130221 2170 tablet_bootstrap.cc:492] T 32895feb7def4132953f5a4d6bf0bbd8 P 75c4f59060974a3b823305b9dd339a7e: No bootstrap required, opened a new log
I20250627 13:52:36.130398 2170 ts_tablet_manager.cc:1397] T 32895feb7def4132953f5a4d6bf0bbd8 P 75c4f59060974a3b823305b9dd339a7e: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20250627 13:52:36.131081 2170 raft_consensus.cc:357] T 32895feb7def4132953f5a4d6bf0bbd8 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:36.131268 2170 raft_consensus.cc:383] T 32895feb7def4132953f5a4d6bf0bbd8 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:36.131343 2170 raft_consensus.cc:738] T 32895feb7def4132953f5a4d6bf0bbd8 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c4f59060974a3b823305b9dd339a7e, State: Initialized, Role: FOLLOWER
I20250627 13:52:36.131552 2170 consensus_queue.cc:260] T 32895feb7def4132953f5a4d6bf0bbd8 P 75c4f59060974a3b823305b9dd339a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:36.131999 2170 ts_tablet_manager.cc:1428] T 32895feb7def4132953f5a4d6bf0bbd8 P 75c4f59060974a3b823305b9dd339a7e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250627 13:52:36.168035 2004 raft_consensus.cc:491] T b9b2432425534c4db188ac324295513d P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:36.168246 2004 raft_consensus.cc:513] T b9b2432425534c4db188ac324295513d P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:36.168872 2004 leader_election.cc:290] T b9b2432425534c4db188ac324295513d P 5c28ff2fe9424379819d478fdd093cfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499), 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68:34289)
I20250627 13:52:36.169287 728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9b2432425534c4db188ac324295513d" candidate_uuid: "5c28ff2fe9424379819d478fdd093cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c4f59060974a3b823305b9dd339a7e" is_pre_election: true
I20250627 13:52:36.169430 992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9b2432425534c4db188ac324295513d" candidate_uuid: "5c28ff2fe9424379819d478fdd093cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" is_pre_election: true
I20250627 13:52:36.169499 728 raft_consensus.cc:2466] T b9b2432425534c4db188ac324295513d P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c28ff2fe9424379819d478fdd093cfc in term 0.
I20250627 13:52:36.169633 992 raft_consensus.cc:2466] T b9b2432425534c4db188ac324295513d P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c28ff2fe9424379819d478fdd093cfc in term 0.
I20250627 13:52:36.170011 1190 leader_election.cc:304] T b9b2432425534c4db188ac324295513d P 5c28ff2fe9424379819d478fdd093cfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c28ff2fe9424379819d478fdd093cfc, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:36.170332 2004 raft_consensus.cc:2802] T b9b2432425534c4db188ac324295513d P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:36.170425 2004 raft_consensus.cc:491] T b9b2432425534c4db188ac324295513d P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:36.170531 2004 raft_consensus.cc:3058] T b9b2432425534c4db188ac324295513d P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:36.172406 2004 raft_consensus.cc:513] T b9b2432425534c4db188ac324295513d P 5c28ff2fe9424379819d478fdd093cfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:36.172925 2004 leader_election.cc:290] T b9b2432425534c4db188ac324295513d P 5c28ff2fe9424379819d478fdd093cfc [CANDIDATE]: Term 1 election: Requested vote from peers 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499), 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68:34289)
I20250627 13:52:36.173300 728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9b2432425534c4db188ac324295513d" candidate_uuid: "5c28ff2fe9424379819d478fdd093cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c4f59060974a3b823305b9dd339a7e"
I20250627 13:52:36.173334 992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9b2432425534c4db188ac324295513d" candidate_uuid: "5c28ff2fe9424379819d478fdd093cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e8c97891d74b38b5c130d3bcbabcb6"
I20250627 13:52:36.173444 728 raft_consensus.cc:3058] T b9b2432425534c4db188ac324295513d P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:36.173489 992 raft_consensus.cc:3058] T b9b2432425534c4db188ac324295513d P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:36.175887 728 raft_consensus.cc:2466] T b9b2432425534c4db188ac324295513d P 75c4f59060974a3b823305b9dd339a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c28ff2fe9424379819d478fdd093cfc in term 1.
I20250627 13:52:36.175886 992 raft_consensus.cc:2466] T b9b2432425534c4db188ac324295513d P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c28ff2fe9424379819d478fdd093cfc in term 1.
I20250627 13:52:36.176301 1190 leader_election.cc:304] T b9b2432425534c4db188ac324295513d P 5c28ff2fe9424379819d478fdd093cfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 50e8c97891d74b38b5c130d3bcbabcb6, 5c28ff2fe9424379819d478fdd093cfc, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:36.176604 2004 raft_consensus.cc:2802] T b9b2432425534c4db188ac324295513d P 5c28ff2fe9424379819d478fdd093cfc [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:36.177042 2004 raft_consensus.cc:695] T b9b2432425534c4db188ac324295513d P 5c28ff2fe9424379819d478fdd093cfc [term 1 LEADER]: Becoming Leader. State: Replica: 5c28ff2fe9424379819d478fdd093cfc, State: Running, Role: LEADER
I20250627 13:52:36.177338 2004 consensus_queue.cc:237] T b9b2432425534c4db188ac324295513d P 5c28ff2fe9424379819d478fdd093cfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:36.180069 454 catalog_manager.cc:5582] T b9b2432425534c4db188ac324295513d P 5c28ff2fe9424379819d478fdd093cfc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70). New cstate: current_term: 1 leader_uuid: "5c28ff2fe9424379819d478fdd093cfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } health_report { overall_health: UNKNOWN } } }
I20250627 13:52:36.201994 2004 raft_consensus.cc:491] T 32895feb7def4132953f5a4d6bf0bbd8 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:36.202154 2004 raft_consensus.cc:513] T 32895feb7def4132953f5a4d6bf0bbd8 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:36.202759 2004 leader_election.cc:290] T 32895feb7def4132953f5a4d6bf0bbd8 P 5c28ff2fe9424379819d478fdd093cfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499), 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68:34289)
I20250627 13:52:36.203217 728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32895feb7def4132953f5a4d6bf0bbd8" candidate_uuid: "5c28ff2fe9424379819d478fdd093cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c4f59060974a3b823305b9dd339a7e" is_pre_election: true
I20250627 13:52:36.203280 992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32895feb7def4132953f5a4d6bf0bbd8" candidate_uuid: "5c28ff2fe9424379819d478fdd093cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" is_pre_election: true
I20250627 13:52:36.203423 728 raft_consensus.cc:2466] T 32895feb7def4132953f5a4d6bf0bbd8 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c28ff2fe9424379819d478fdd093cfc in term 0.
I20250627 13:52:36.203534 992 raft_consensus.cc:2466] T 32895feb7def4132953f5a4d6bf0bbd8 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c28ff2fe9424379819d478fdd093cfc in term 0.
I20250627 13:52:36.203848 1190 leader_election.cc:304] T 32895feb7def4132953f5a4d6bf0bbd8 P 5c28ff2fe9424379819d478fdd093cfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c28ff2fe9424379819d478fdd093cfc, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:36.204085 2004 raft_consensus.cc:2802] T 32895feb7def4132953f5a4d6bf0bbd8 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:36.204178 2004 raft_consensus.cc:491] T 32895feb7def4132953f5a4d6bf0bbd8 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:36.204238 2004 raft_consensus.cc:3058] T 32895feb7def4132953f5a4d6bf0bbd8 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:36.206250 2004 raft_consensus.cc:513] T 32895feb7def4132953f5a4d6bf0bbd8 P 5c28ff2fe9424379819d478fdd093cfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:36.206882 2004 leader_election.cc:290] T 32895feb7def4132953f5a4d6bf0bbd8 P 5c28ff2fe9424379819d478fdd093cfc [CANDIDATE]: Term 1 election: Requested vote from peers 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499), 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68:34289)
I20250627 13:52:36.207262 728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32895feb7def4132953f5a4d6bf0bbd8" candidate_uuid: "5c28ff2fe9424379819d478fdd093cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c4f59060974a3b823305b9dd339a7e"
I20250627 13:52:36.207404 992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32895feb7def4132953f5a4d6bf0bbd8" candidate_uuid: "5c28ff2fe9424379819d478fdd093cfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e8c97891d74b38b5c130d3bcbabcb6"
I20250627 13:52:36.207471 728 raft_consensus.cc:3058] T 32895feb7def4132953f5a4d6bf0bbd8 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:36.207607 992 raft_consensus.cc:3058] T 32895feb7def4132953f5a4d6bf0bbd8 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:36.209338 728 raft_consensus.cc:2466] T 32895feb7def4132953f5a4d6bf0bbd8 P 75c4f59060974a3b823305b9dd339a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c28ff2fe9424379819d478fdd093cfc in term 1.
I20250627 13:52:36.209482 992 raft_consensus.cc:2466] T 32895feb7def4132953f5a4d6bf0bbd8 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c28ff2fe9424379819d478fdd093cfc in term 1.
I20250627 13:52:36.209807 1190 leader_election.cc:304] T 32895feb7def4132953f5a4d6bf0bbd8 P 5c28ff2fe9424379819d478fdd093cfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c28ff2fe9424379819d478fdd093cfc, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:36.210104 2004 raft_consensus.cc:2802] T 32895feb7def4132953f5a4d6bf0bbd8 P 5c28ff2fe9424379819d478fdd093cfc [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:36.210239 2004 raft_consensus.cc:695] T 32895feb7def4132953f5a4d6bf0bbd8 P 5c28ff2fe9424379819d478fdd093cfc [term 1 LEADER]: Becoming Leader. State: Replica: 5c28ff2fe9424379819d478fdd093cfc, State: Running, Role: LEADER
I20250627 13:52:36.210489 2004 consensus_queue.cc:237] T 32895feb7def4132953f5a4d6bf0bbd8 P 5c28ff2fe9424379819d478fdd093cfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } }
I20250627 13:52:36.213198 454 catalog_manager.cc:5582] T 32895feb7def4132953f5a4d6bf0bbd8 P 5c28ff2fe9424379819d478fdd093cfc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70). New cstate: current_term: 1 leader_uuid: "5c28ff2fe9424379819d478fdd093cfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } health_report { overall_health: HEALTHY } } }
I20250627 13:52:36.334076 1890 raft_consensus.cc:491] T 574d86438dc240aca8fd3791c70d19b3 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:36.334254 1890 raft_consensus.cc:513] T 574d86438dc240aca8fd3791c70d19b3 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:36.335040 1890 leader_election.cc:290] T 574d86438dc240aca8fd3791c70d19b3 P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023), fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747)
I20250627 13:52:36.335740 597 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "574d86438dc240aca8fd3791c70d19b3" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98" is_pre_election: true
I20250627 13:52:36.335966 597 raft_consensus.cc:2466] T 574d86438dc240aca8fd3791c70d19b3 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 0.
I20250627 13:52:36.335918 861 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "574d86438dc240aca8fd3791c70d19b3" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e" is_pre_election: true
I20250627 13:52:36.336169 861 raft_consensus.cc:2466] T 574d86438dc240aca8fd3791c70d19b3 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 0.
I20250627 13:52:36.336400 1056 leader_election.cc:304] T 574d86438dc240aca8fd3791c70d19b3 P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, 9f767210a91f44ae8b83b994131eee98; no voters:
I20250627 13:52:36.336658 1890 raft_consensus.cc:2802] T 574d86438dc240aca8fd3791c70d19b3 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:36.336764 1890 raft_consensus.cc:491] T 574d86438dc240aca8fd3791c70d19b3 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:36.336855 1890 raft_consensus.cc:3058] T 574d86438dc240aca8fd3791c70d19b3 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:36.338668 1890 raft_consensus.cc:513] T 574d86438dc240aca8fd3791c70d19b3 P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:36.339191 1890 leader_election.cc:290] T 574d86438dc240aca8fd3791c70d19b3 P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 election: Requested vote from peers 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023), fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747)
I20250627 13:52:36.339596 597 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "574d86438dc240aca8fd3791c70d19b3" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98"
I20250627 13:52:36.339666 861 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "574d86438dc240aca8fd3791c70d19b3" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e"
I20250627 13:52:36.339767 597 raft_consensus.cc:3058] T 574d86438dc240aca8fd3791c70d19b3 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:36.339856 861 raft_consensus.cc:3058] T 574d86438dc240aca8fd3791c70d19b3 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:36.341802 597 raft_consensus.cc:2466] T 574d86438dc240aca8fd3791c70d19b3 P 9f767210a91f44ae8b83b994131eee98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 1.
I20250627 13:52:36.342142 1056 leader_election.cc:304] T 574d86438dc240aca8fd3791c70d19b3 P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, 9f767210a91f44ae8b83b994131eee98; no voters:
I20250627 13:52:36.342406 1890 raft_consensus.cc:2802] T 574d86438dc240aca8fd3791c70d19b3 P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:36.342523 1890 raft_consensus.cc:695] T 574d86438dc240aca8fd3791c70d19b3 P 957a8a8be3f542f48e8824e581fd0f89 [term 1 LEADER]: Becoming Leader. State: Replica: 957a8a8be3f542f48e8824e581fd0f89, State: Running, Role: LEADER
I20250627 13:52:36.342610 861 raft_consensus.cc:2466] T 574d86438dc240aca8fd3791c70d19b3 P fa66786862da49ae826fb519c7d89b5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 1.
I20250627 13:52:36.342790 1890 consensus_queue.cc:237] T 574d86438dc240aca8fd3791c70d19b3 P 957a8a8be3f542f48e8824e581fd0f89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:36.345788 454 catalog_manager.cc:5582] T 574d86438dc240aca8fd3791c70d19b3 P 957a8a8be3f542f48e8824e581fd0f89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69). New cstate: current_term: 1 leader_uuid: "957a8a8be3f542f48e8824e581fd0f89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } health_report { overall_health: HEALTHY } } }
I20250627 13:52:36.599221 2004 consensus_queue.cc:1035] T 32895feb7def4132953f5a4d6bf0bbd8 P 5c28ff2fe9424379819d478fdd093cfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:36.620456 2185 consensus_queue.cc:1035] T 32895feb7def4132953f5a4d6bf0bbd8 P 5c28ff2fe9424379819d478fdd093cfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:36.657299 2004 consensus_queue.cc:1035] T b9b2432425534c4db188ac324295513d P 5c28ff2fe9424379819d478fdd093cfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:36.663923 2194 consensus_queue.cc:1035] T b9b2432425534c4db188ac324295513d P 5c28ff2fe9424379819d478fdd093cfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250627 13:52:36.759393 2190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:36.759841 2190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:36.768558 2190 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20250627 13:52:36.891433 2224 consensus_queue.cc:1035] T 574d86438dc240aca8fd3791c70d19b3 P 957a8a8be3f542f48e8824e581fd0f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:36.900367 2224 consensus_queue.cc:1035] T 574d86438dc240aca8fd3791c70d19b3 P 957a8a8be3f542f48e8824e581fd0f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 245.414 ms
time per row: 0.122707 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20250627 13:52:38.191102 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49852:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250627 13:52:38.200822 578 tablet_service.cc:1468] Processing CreateTablet for tablet 0118ccc5093e4971a48f0270c2224959 (DEFAULT_TABLE table=rebalance_test_table_10 [id=4d00aa77c1f745feabe07e2fff678e99]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:38.200942 841 tablet_service.cc:1468] Processing CreateTablet for tablet 0118ccc5093e4971a48f0270c2224959 (DEFAULT_TABLE table=rebalance_test_table_10 [id=4d00aa77c1f745feabe07e2fff678e99]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:38.201355 578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0118ccc5093e4971a48f0270c2224959. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:38.201419 841 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0118ccc5093e4971a48f0270c2224959. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:38.202452 840 tablet_service.cc:1468] Processing CreateTablet for tablet b153e10a78914a84a5bd7095fcc00394 (DEFAULT_TABLE table=rebalance_test_table_10 [id=4d00aa77c1f745feabe07e2fff678e99]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:38.202631 577 tablet_service.cc:1468] Processing CreateTablet for tablet b153e10a78914a84a5bd7095fcc00394 (DEFAULT_TABLE table=rebalance_test_table_10 [id=4d00aa77c1f745feabe07e2fff678e99]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:38.202853 840 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b153e10a78914a84a5bd7095fcc00394. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:38.203019 577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b153e10a78914a84a5bd7095fcc00394. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:38.205332 710 tablet_service.cc:1468] Processing CreateTablet for tablet 37e4e8562d25473db32d95236b51ee8a (DEFAULT_TABLE table=rebalance_test_table_10 [id=4d00aa77c1f745feabe07e2fff678e99]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:38.205760 710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37e4e8562d25473db32d95236b51ee8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:38.207005 971 tablet_service.cc:1468] Processing CreateTablet for tablet 37e4e8562d25473db32d95236b51ee8a (DEFAULT_TABLE table=rebalance_test_table_10 [id=4d00aa77c1f745feabe07e2fff678e99]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:38.207451 971 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37e4e8562d25473db32d95236b51ee8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:38.209905 1103 tablet_service.cc:1468] Processing CreateTablet for tablet 0118ccc5093e4971a48f0270c2224959 (DEFAULT_TABLE table=rebalance_test_table_10 [id=4d00aa77c1f745feabe07e2fff678e99]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:38.210450 1103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0118ccc5093e4971a48f0270c2224959. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:38.211323 1102 tablet_service.cc:1468] Processing CreateTablet for tablet b153e10a78914a84a5bd7095fcc00394 (DEFAULT_TABLE table=rebalance_test_table_10 [id=4d00aa77c1f745feabe07e2fff678e99]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:38.211691 1102 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b153e10a78914a84a5bd7095fcc00394. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:38.214759 1233 tablet_service.cc:1468] Processing CreateTablet for tablet 37e4e8562d25473db32d95236b51ee8a (DEFAULT_TABLE table=rebalance_test_table_10 [id=4d00aa77c1f745feabe07e2fff678e99]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:38.215289 1233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37e4e8562d25473db32d95236b51ee8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:38.229347 2263 tablet_bootstrap.cc:492] T b153e10a78914a84a5bd7095fcc00394 P 957a8a8be3f542f48e8824e581fd0f89: Bootstrap starting.
I20250627 13:52:38.233716 2262 tablet_bootstrap.cc:492] T b153e10a78914a84a5bd7095fcc00394 P 9f767210a91f44ae8b83b994131eee98: Bootstrap starting.
I20250627 13:52:38.234633 2263 tablet_bootstrap.cc:654] T b153e10a78914a84a5bd7095fcc00394 P 957a8a8be3f542f48e8824e581fd0f89: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:38.236734 2262 tablet_bootstrap.cc:654] T b153e10a78914a84a5bd7095fcc00394 P 9f767210a91f44ae8b83b994131eee98: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:38.239743 2261 tablet_bootstrap.cc:492] T 37e4e8562d25473db32d95236b51ee8a P 50e8c97891d74b38b5c130d3bcbabcb6: Bootstrap starting.
I20250627 13:52:38.240135 2265 tablet_bootstrap.cc:492] T 37e4e8562d25473db32d95236b51ee8a P 75c4f59060974a3b823305b9dd339a7e: Bootstrap starting.
I20250627 13:52:38.240506 2260 tablet_bootstrap.cc:492] T b153e10a78914a84a5bd7095fcc00394 P fa66786862da49ae826fb519c7d89b5e: Bootstrap starting.
I20250627 13:52:38.241925 2265 tablet_bootstrap.cc:654] T 37e4e8562d25473db32d95236b51ee8a P 75c4f59060974a3b823305b9dd339a7e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:38.247576 2260 tablet_bootstrap.cc:654] T b153e10a78914a84a5bd7095fcc00394 P fa66786862da49ae826fb519c7d89b5e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:38.246556 2261 tablet_bootstrap.cc:654] T 37e4e8562d25473db32d95236b51ee8a P 50e8c97891d74b38b5c130d3bcbabcb6: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:38.252219 2267 tablet_bootstrap.cc:492] T 37e4e8562d25473db32d95236b51ee8a P 5c28ff2fe9424379819d478fdd093cfc: Bootstrap starting.
I20250627 13:52:38.253449 2265 tablet_bootstrap.cc:492] T 37e4e8562d25473db32d95236b51ee8a P 75c4f59060974a3b823305b9dd339a7e: No bootstrap required, opened a new log
I20250627 13:52:38.253697 2265 ts_tablet_manager.cc:1397] T 37e4e8562d25473db32d95236b51ee8a P 75c4f59060974a3b823305b9dd339a7e: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20250627 13:52:38.254218 2261 tablet_bootstrap.cc:492] T 37e4e8562d25473db32d95236b51ee8a P 50e8c97891d74b38b5c130d3bcbabcb6: No bootstrap required, opened a new log
I20250627 13:52:38.254377 2261 ts_tablet_manager.cc:1397] T 37e4e8562d25473db32d95236b51ee8a P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.004s
I20250627 13:52:38.254611 2260 tablet_bootstrap.cc:492] T b153e10a78914a84a5bd7095fcc00394 P fa66786862da49ae826fb519c7d89b5e: No bootstrap required, opened a new log
I20250627 13:52:38.254886 2265 raft_consensus.cc:357] T 37e4e8562d25473db32d95236b51ee8a P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:38.255241 2265 raft_consensus.cc:383] T 37e4e8562d25473db32d95236b51ee8a P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:38.255177 2261 raft_consensus.cc:357] T 37e4e8562d25473db32d95236b51ee8a P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:38.255378 2261 raft_consensus.cc:383] T 37e4e8562d25473db32d95236b51ee8a P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:38.255368 2265 raft_consensus.cc:738] T 37e4e8562d25473db32d95236b51ee8a P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c4f59060974a3b823305b9dd339a7e, State: Initialized, Role: FOLLOWER
I20250627 13:52:38.255474 2261 raft_consensus.cc:738] T 37e4e8562d25473db32d95236b51ee8a P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e8c97891d74b38b5c130d3bcbabcb6, State: Initialized, Role: FOLLOWER
I20250627 13:52:38.255719 2265 consensus_queue.cc:260] T 37e4e8562d25473db32d95236b51ee8a P 75c4f59060974a3b823305b9dd339a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:38.256086 2260 ts_tablet_manager.cc:1397] T b153e10a78914a84a5bd7095fcc00394 P fa66786862da49ae826fb519c7d89b5e: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.003s
I20250627 13:52:38.256314 2265 ts_tablet_manager.cc:1428] T 37e4e8562d25473db32d95236b51ee8a P 75c4f59060974a3b823305b9dd339a7e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:52:38.256870 2262 tablet_bootstrap.cc:492] T b153e10a78914a84a5bd7095fcc00394 P 9f767210a91f44ae8b83b994131eee98: No bootstrap required, opened a new log
I20250627 13:52:38.256915 2260 raft_consensus.cc:357] T b153e10a78914a84a5bd7095fcc00394 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:38.257038 2262 ts_tablet_manager.cc:1397] T b153e10a78914a84a5bd7095fcc00394 P 9f767210a91f44ae8b83b994131eee98: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.005s
I20250627 13:52:38.257153 2260 raft_consensus.cc:383] T b153e10a78914a84a5bd7095fcc00394 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:38.257480 2263 tablet_bootstrap.cc:492] T b153e10a78914a84a5bd7095fcc00394 P 957a8a8be3f542f48e8824e581fd0f89: No bootstrap required, opened a new log
I20250627 13:52:38.257683 2263 ts_tablet_manager.cc:1397] T b153e10a78914a84a5bd7095fcc00394 P 957a8a8be3f542f48e8824e581fd0f89: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.016s
I20250627 13:52:38.258054 2262 raft_consensus.cc:357] T b153e10a78914a84a5bd7095fcc00394 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:38.258309 2262 raft_consensus.cc:383] T b153e10a78914a84a5bd7095fcc00394 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:38.258430 2262 raft_consensus.cc:738] T b153e10a78914a84a5bd7095fcc00394 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f767210a91f44ae8b83b994131eee98, State: Initialized, Role: FOLLOWER
I20250627 13:52:38.258534 2263 raft_consensus.cc:357] T b153e10a78914a84a5bd7095fcc00394 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:38.258718 2263 raft_consensus.cc:383] T b153e10a78914a84a5bd7095fcc00394 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:38.258659 2262 consensus_queue.cc:260] T b153e10a78914a84a5bd7095fcc00394 P 9f767210a91f44ae8b83b994131eee98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:38.258833 2263 raft_consensus.cc:738] T b153e10a78914a84a5bd7095fcc00394 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957a8a8be3f542f48e8824e581fd0f89, State: Initialized, Role: FOLLOWER
I20250627 13:52:38.259074 2263 consensus_queue.cc:260] T b153e10a78914a84a5bd7095fcc00394 P 957a8a8be3f542f48e8824e581fd0f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:38.259213 2262 ts_tablet_manager.cc:1428] T b153e10a78914a84a5bd7095fcc00394 P 9f767210a91f44ae8b83b994131eee98: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:52:38.259495 2262 tablet_bootstrap.cc:492] T 0118ccc5093e4971a48f0270c2224959 P 9f767210a91f44ae8b83b994131eee98: Bootstrap starting.
I20250627 13:52:38.260499 2263 ts_tablet_manager.cc:1428] T b153e10a78914a84a5bd7095fcc00394 P 957a8a8be3f542f48e8824e581fd0f89: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250627 13:52:38.260816 2263 tablet_bootstrap.cc:492] T 0118ccc5093e4971a48f0270c2224959 P 957a8a8be3f542f48e8824e581fd0f89: Bootstrap starting.
I20250627 13:52:38.261457 2262 tablet_bootstrap.cc:654] T 0118ccc5093e4971a48f0270c2224959 P 9f767210a91f44ae8b83b994131eee98: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:38.261703 2261 consensus_queue.cc:260] T 37e4e8562d25473db32d95236b51ee8a P 50e8c97891d74b38b5c130d3bcbabcb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:38.262507 2263 tablet_bootstrap.cc:654] T 0118ccc5093e4971a48f0270c2224959 P 957a8a8be3f542f48e8824e581fd0f89: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:38.263110 2261 ts_tablet_manager.cc:1428] T 37e4e8562d25473db32d95236b51ee8a P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250627 13:52:38.268875 2093 raft_consensus.cc:491] T b153e10a78914a84a5bd7095fcc00394 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:38.269039 2093 raft_consensus.cc:513] T b153e10a78914a84a5bd7095fcc00394 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:38.269718 2093 leader_election.cc:290] T b153e10a78914a84a5bd7095fcc00394 P 9f767210a91f44ae8b83b994131eee98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747), 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69:42325)
I20250627 13:52:38.287755 2267 tablet_bootstrap.cc:654] T 37e4e8562d25473db32d95236b51ee8a P 5c28ff2fe9424379819d478fdd093cfc: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:38.270306 861 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b153e10a78914a84a5bd7095fcc00394" candidate_uuid: "9f767210a91f44ae8b83b994131eee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e" is_pre_election: true
I20250627 13:52:38.288010 2260 raft_consensus.cc:738] T b153e10a78914a84a5bd7095fcc00394 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa66786862da49ae826fb519c7d89b5e, State: Initialized, Role: FOLLOWER
I20250627 13:52:38.288264 2260 consensus_queue.cc:260] T b153e10a78914a84a5bd7095fcc00394 P fa66786862da49ae826fb519c7d89b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:38.289629 861 raft_consensus.cc:2466] T b153e10a78914a84a5bd7095fcc00394 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f767210a91f44ae8b83b994131eee98 in term 0.
I20250627 13:52:38.290525 533 leader_election.cc:304] T b153e10a78914a84a5bd7095fcc00394 P 9f767210a91f44ae8b83b994131eee98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f767210a91f44ae8b83b994131eee98, fa66786862da49ae826fb519c7d89b5e; no voters:
I20250627 13:52:38.291549 2260 ts_tablet_manager.cc:1428] T b153e10a78914a84a5bd7095fcc00394 P fa66786862da49ae826fb519c7d89b5e: Time spent starting tablet: real 0.035s user 0.001s sys 0.001s
I20250627 13:52:38.298388 2260 tablet_bootstrap.cc:492] T 0118ccc5093e4971a48f0270c2224959 P fa66786862da49ae826fb519c7d89b5e: Bootstrap starting.
I20250627 13:52:38.302706 2093 raft_consensus.cc:2802] T b153e10a78914a84a5bd7095fcc00394 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:38.302850 2093 raft_consensus.cc:491] T b153e10a78914a84a5bd7095fcc00394 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:38.302949 2093 raft_consensus.cc:3058] T b153e10a78914a84a5bd7095fcc00394 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:38.305037 2093 raft_consensus.cc:513] T b153e10a78914a84a5bd7095fcc00394 P 9f767210a91f44ae8b83b994131eee98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:38.305822 2093 leader_election.cc:290] T b153e10a78914a84a5bd7095fcc00394 P 9f767210a91f44ae8b83b994131eee98 [CANDIDATE]: Term 1 election: Requested vote from peers fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747), 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69:42325)
I20250627 13:52:38.306685 1122 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b153e10a78914a84a5bd7095fcc00394" candidate_uuid: "9f767210a91f44ae8b83b994131eee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957a8a8be3f542f48e8824e581fd0f89"
I20250627 13:52:38.306880 1123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b153e10a78914a84a5bd7095fcc00394" candidate_uuid: "9f767210a91f44ae8b83b994131eee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957a8a8be3f542f48e8824e581fd0f89" is_pre_election: true
I20250627 13:52:38.306927 1122 raft_consensus.cc:3058] T b153e10a78914a84a5bd7095fcc00394 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:38.308770 2267 tablet_bootstrap.cc:492] T 37e4e8562d25473db32d95236b51ee8a P 5c28ff2fe9424379819d478fdd093cfc: No bootstrap required, opened a new log
I20250627 13:52:38.308938 2267 ts_tablet_manager.cc:1397] T 37e4e8562d25473db32d95236b51ee8a P 5c28ff2fe9424379819d478fdd093cfc: Time spent bootstrapping tablet: real 0.057s user 0.004s sys 0.000s
I20250627 13:52:38.309847 1122 raft_consensus.cc:2466] T b153e10a78914a84a5bd7095fcc00394 P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f767210a91f44ae8b83b994131eee98 in term 1.
I20250627 13:52:38.310283 534 leader_election.cc:304] T b153e10a78914a84a5bd7095fcc00394 P 9f767210a91f44ae8b83b994131eee98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, 9f767210a91f44ae8b83b994131eee98; no voters:
I20250627 13:52:38.310575 2273 raft_consensus.cc:2802] T b153e10a78914a84a5bd7095fcc00394 P 9f767210a91f44ae8b83b994131eee98 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:38.310742 2273 raft_consensus.cc:695] T b153e10a78914a84a5bd7095fcc00394 P 9f767210a91f44ae8b83b994131eee98 [term 1 LEADER]: Becoming Leader. State: Replica: 9f767210a91f44ae8b83b994131eee98, State: Running, Role: LEADER
I20250627 13:52:38.311246 2260 tablet_bootstrap.cc:654] T 0118ccc5093e4971a48f0270c2224959 P fa66786862da49ae826fb519c7d89b5e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:38.311583 2273 consensus_queue.cc:237] T b153e10a78914a84a5bd7095fcc00394 P 9f767210a91f44ae8b83b994131eee98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } }
I20250627 13:52:38.311301 861 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b153e10a78914a84a5bd7095fcc00394" candidate_uuid: "9f767210a91f44ae8b83b994131eee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e"
I20250627 13:52:38.317844 861 raft_consensus.cc:3058] T b153e10a78914a84a5bd7095fcc00394 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:38.321429 861 raft_consensus.cc:2466] T b153e10a78914a84a5bd7095fcc00394 P fa66786862da49ae826fb519c7d89b5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f767210a91f44ae8b83b994131eee98 in term 1.
I20250627 13:52:38.322227 2267 raft_consensus.cc:357] T 37e4e8562d25473db32d95236b51ee8a P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:38.322710 2267 raft_consensus.cc:383] T 37e4e8562d25473db32d95236b51ee8a P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:38.324218 2267 raft_consensus.cc:738] T 37e4e8562d25473db32d95236b51ee8a P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c28ff2fe9424379819d478fdd093cfc, State: Initialized, Role: FOLLOWER
I20250627 13:52:38.324558 2267 consensus_queue.cc:260] T 37e4e8562d25473db32d95236b51ee8a P 5c28ff2fe9424379819d478fdd093cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:38.325181 451 catalog_manager.cc:5582] T b153e10a78914a84a5bd7095fcc00394 P 9f767210a91f44ae8b83b994131eee98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f767210a91f44ae8b83b994131eee98 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "9f767210a91f44ae8b83b994131eee98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } health_report { overall_health: HEALTHY } } }
I20250627 13:52:38.326337 2267 ts_tablet_manager.cc:1428] T 37e4e8562d25473db32d95236b51ee8a P 5c28ff2fe9424379819d478fdd093cfc: Time spent starting tablet: real 0.017s user 0.003s sys 0.000s
I20250627 13:52:38.327869 2262 tablet_bootstrap.cc:492] T 0118ccc5093e4971a48f0270c2224959 P 9f767210a91f44ae8b83b994131eee98: No bootstrap required, opened a new log
I20250627 13:52:38.328034 2262 ts_tablet_manager.cc:1397] T 0118ccc5093e4971a48f0270c2224959 P 9f767210a91f44ae8b83b994131eee98: Time spent bootstrapping tablet: real 0.069s user 0.018s sys 0.042s
I20250627 13:52:38.328784 2262 raft_consensus.cc:357] T 0118ccc5093e4971a48f0270c2224959 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:38.328990 2262 raft_consensus.cc:383] T 0118ccc5093e4971a48f0270c2224959 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:38.329062 2262 raft_consensus.cc:738] T 0118ccc5093e4971a48f0270c2224959 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f767210a91f44ae8b83b994131eee98, State: Initialized, Role: FOLLOWER
I20250627 13:52:38.330325 2263 tablet_bootstrap.cc:492] T 0118ccc5093e4971a48f0270c2224959 P 957a8a8be3f542f48e8824e581fd0f89: No bootstrap required, opened a new log
I20250627 13:52:38.330555 2263 ts_tablet_manager.cc:1397] T 0118ccc5093e4971a48f0270c2224959 P 957a8a8be3f542f48e8824e581fd0f89: Time spent bootstrapping tablet: real 0.070s user 0.020s sys 0.025s
I20250627 13:52:38.331337 2263 raft_consensus.cc:357] T 0118ccc5093e4971a48f0270c2224959 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:38.331530 2263 raft_consensus.cc:383] T 0118ccc5093e4971a48f0270c2224959 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:38.331593 2263 raft_consensus.cc:738] T 0118ccc5093e4971a48f0270c2224959 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957a8a8be3f542f48e8824e581fd0f89, State: Initialized, Role: FOLLOWER
I20250627 13:52:38.331844 2263 consensus_queue.cc:260] T 0118ccc5093e4971a48f0270c2224959 P 957a8a8be3f542f48e8824e581fd0f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:38.332329 2263 ts_tablet_manager.cc:1428] T 0118ccc5093e4971a48f0270c2224959 P 957a8a8be3f542f48e8824e581fd0f89: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250627 13:52:38.329339 2262 consensus_queue.cc:260] T 0118ccc5093e4971a48f0270c2224959 P 9f767210a91f44ae8b83b994131eee98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:38.332911 2262 ts_tablet_manager.cc:1428] T 0118ccc5093e4971a48f0270c2224959 P 9f767210a91f44ae8b83b994131eee98: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250627 13:52:38.334648 2260 tablet_bootstrap.cc:492] T 0118ccc5093e4971a48f0270c2224959 P fa66786862da49ae826fb519c7d89b5e: No bootstrap required, opened a new log
I20250627 13:52:38.334882 2260 ts_tablet_manager.cc:1397] T 0118ccc5093e4971a48f0270c2224959 P fa66786862da49ae826fb519c7d89b5e: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.003s
I20250627 13:52:38.335731 2260 raft_consensus.cc:357] T 0118ccc5093e4971a48f0270c2224959 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:38.335954 2260 raft_consensus.cc:383] T 0118ccc5093e4971a48f0270c2224959 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:38.336031 2260 raft_consensus.cc:738] T 0118ccc5093e4971a48f0270c2224959 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa66786862da49ae826fb519c7d89b5e, State: Initialized, Role: FOLLOWER
I20250627 13:52:38.336324 2260 consensus_queue.cc:260] T 0118ccc5093e4971a48f0270c2224959 P fa66786862da49ae826fb519c7d89b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:38.336848 2260 ts_tablet_manager.cc:1428] T 0118ccc5093e4971a48f0270c2224959 P fa66786862da49ae826fb519c7d89b5e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250627 13:52:38.417286 2158 raft_consensus.cc:491] T 37e4e8562d25473db32d95236b51ee8a P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:38.417462 2158 raft_consensus.cc:513] T 37e4e8562d25473db32d95236b51ee8a P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:38.418046 2158 leader_election.cc:290] T 37e4e8562d25473db32d95236b51ee8a P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70:39887), 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499)
I20250627 13:52:38.418537 728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37e4e8562d25473db32d95236b51ee8a" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c4f59060974a3b823305b9dd339a7e" is_pre_election: true
I20250627 13:52:38.418524 1253 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37e4e8562d25473db32d95236b51ee8a" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c28ff2fe9424379819d478fdd093cfc" is_pre_election: true
I20250627 13:52:38.418736 1253 raft_consensus.cc:2466] T 37e4e8562d25473db32d95236b51ee8a P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 0.
I20250627 13:52:38.418745 728 raft_consensus.cc:2466] T 37e4e8562d25473db32d95236b51ee8a P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 0.
I20250627 13:52:38.419174 928 leader_election.cc:304] T 37e4e8562d25473db32d95236b51ee8a P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 50e8c97891d74b38b5c130d3bcbabcb6, 5c28ff2fe9424379819d478fdd093cfc, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:38.419454 2158 raft_consensus.cc:2802] T 37e4e8562d25473db32d95236b51ee8a P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:38.419572 2158 raft_consensus.cc:491] T 37e4e8562d25473db32d95236b51ee8a P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:38.419648 2158 raft_consensus.cc:3058] T 37e4e8562d25473db32d95236b51ee8a P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:38.421556 2158 raft_consensus.cc:513] T 37e4e8562d25473db32d95236b51ee8a P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:38.422034 2158 leader_election.cc:290] T 37e4e8562d25473db32d95236b51ee8a P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 election: Requested vote from peers 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70:39887), 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499)
I20250627 13:52:38.422421 1253 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37e4e8562d25473db32d95236b51ee8a" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c28ff2fe9424379819d478fdd093cfc"
I20250627 13:52:38.422582 1253 raft_consensus.cc:3058] T 37e4e8562d25473db32d95236b51ee8a P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:38.422506 728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37e4e8562d25473db32d95236b51ee8a" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c4f59060974a3b823305b9dd339a7e"
I20250627 13:52:38.422735 728 raft_consensus.cc:3058] T 37e4e8562d25473db32d95236b51ee8a P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:38.424489 1253 raft_consensus.cc:2466] T 37e4e8562d25473db32d95236b51ee8a P 5c28ff2fe9424379819d478fdd093cfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 1.
I20250627 13:52:38.424525 728 raft_consensus.cc:2466] T 37e4e8562d25473db32d95236b51ee8a P 75c4f59060974a3b823305b9dd339a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 1.
I20250627 13:52:38.424890 926 leader_election.cc:304] T 37e4e8562d25473db32d95236b51ee8a P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 50e8c97891d74b38b5c130d3bcbabcb6, 5c28ff2fe9424379819d478fdd093cfc, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:38.425148 2158 raft_consensus.cc:2802] T 37e4e8562d25473db32d95236b51ee8a P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:38.425482 2158 raft_consensus.cc:695] T 37e4e8562d25473db32d95236b51ee8a P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 LEADER]: Becoming Leader. State: Replica: 50e8c97891d74b38b5c130d3bcbabcb6, State: Running, Role: LEADER
I20250627 13:52:38.425776 2158 consensus_queue.cc:237] T 37e4e8562d25473db32d95236b51ee8a P 50e8c97891d74b38b5c130d3bcbabcb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:38.428596 454 catalog_manager.cc:5582] T 37e4e8562d25473db32d95236b51ee8a P 50e8c97891d74b38b5c130d3bcbabcb6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68). New cstate: current_term: 1 leader_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } health_report { overall_health: HEALTHY } } }
I20250627 13:52:38.445091 2249 raft_consensus.cc:491] T 0118ccc5093e4971a48f0270c2224959 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:38.445286 2249 raft_consensus.cc:513] T 0118ccc5093e4971a48f0270c2224959 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:38.445920 2249 leader_election.cc:290] T 0118ccc5093e4971a48f0270c2224959 P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023), fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747)
I20250627 13:52:38.446393 861 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0118ccc5093e4971a48f0270c2224959" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e" is_pre_election: true
I20250627 13:52:38.446393 598 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0118ccc5093e4971a48f0270c2224959" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98" is_pre_election: true
I20250627 13:52:38.446669 861 raft_consensus.cc:2466] T 0118ccc5093e4971a48f0270c2224959 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 0.
I20250627 13:52:38.446688 598 raft_consensus.cc:2466] T 0118ccc5093e4971a48f0270c2224959 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 0.
I20250627 13:52:38.447151 1056 leader_election.cc:304] T 0118ccc5093e4971a48f0270c2224959 P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, 9f767210a91f44ae8b83b994131eee98, fa66786862da49ae826fb519c7d89b5e; no voters:
I20250627 13:52:38.447434 2249 raft_consensus.cc:2802] T 0118ccc5093e4971a48f0270c2224959 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:38.447532 2249 raft_consensus.cc:491] T 0118ccc5093e4971a48f0270c2224959 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:38.447620 2249 raft_consensus.cc:3058] T 0118ccc5093e4971a48f0270c2224959 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:38.449466 2249 raft_consensus.cc:513] T 0118ccc5093e4971a48f0270c2224959 P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:38.449934 2249 leader_election.cc:290] T 0118ccc5093e4971a48f0270c2224959 P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 election: Requested vote from peers 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023), fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747)
I20250627 13:52:38.450421 598 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0118ccc5093e4971a48f0270c2224959" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98"
I20250627 13:52:38.450421 861 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0118ccc5093e4971a48f0270c2224959" candidate_uuid: "957a8a8be3f542f48e8824e581fd0f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e"
I20250627 13:52:38.450630 598 raft_consensus.cc:3058] T 0118ccc5093e4971a48f0270c2224959 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:38.450630 861 raft_consensus.cc:3058] T 0118ccc5093e4971a48f0270c2224959 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:38.453527 861 raft_consensus.cc:2466] T 0118ccc5093e4971a48f0270c2224959 P fa66786862da49ae826fb519c7d89b5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 1.
I20250627 13:52:38.453527 598 raft_consensus.cc:2466] T 0118ccc5093e4971a48f0270c2224959 P 9f767210a91f44ae8b83b994131eee98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 957a8a8be3f542f48e8824e581fd0f89 in term 1.
I20250627 13:52:38.453944 1058 leader_election.cc:304] T 0118ccc5093e4971a48f0270c2224959 P 957a8a8be3f542f48e8824e581fd0f89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, fa66786862da49ae826fb519c7d89b5e; no voters:
I20250627 13:52:38.454239 2249 raft_consensus.cc:2802] T 0118ccc5093e4971a48f0270c2224959 P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:38.454604 2249 raft_consensus.cc:695] T 0118ccc5093e4971a48f0270c2224959 P 957a8a8be3f542f48e8824e581fd0f89 [term 1 LEADER]: Becoming Leader. State: Replica: 957a8a8be3f542f48e8824e581fd0f89, State: Running, Role: LEADER
I20250627 13:52:38.454905 2249 consensus_queue.cc:237] T 0118ccc5093e4971a48f0270c2224959 P 957a8a8be3f542f48e8824e581fd0f89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:38.457754 454 catalog_manager.cc:5582] T 0118ccc5093e4971a48f0270c2224959 P 957a8a8be3f542f48e8824e581fd0f89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69). New cstate: current_term: 1 leader_uuid: "957a8a8be3f542f48e8824e581fd0f89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } health_report { overall_health: HEALTHY } } }
W20250627 13:52:38.842898 2282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:38.843235 2282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:38.851655 2282 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250627 13:52:38.858655 2273 consensus_queue.cc:1035] T b153e10a78914a84a5bd7095fcc00394 P 9f767210a91f44ae8b83b994131eee98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:38.873781 2249 consensus_queue.cc:1035] T 0118ccc5093e4971a48f0270c2224959 P 957a8a8be3f542f48e8824e581fd0f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:38.880604 2287 consensus_queue.cc:1035] T b153e10a78914a84a5bd7095fcc00394 P 9f767210a91f44ae8b83b994131eee98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:38.882599 2279 consensus_queue.cc:1035] T 0118ccc5093e4971a48f0270c2224959 P 957a8a8be3f542f48e8824e581fd0f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:38.903755 2158 consensus_queue.cc:1035] T 37e4e8562d25473db32d95236b51ee8a P 50e8c97891d74b38b5c130d3bcbabcb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:38.913157 2277 consensus_queue.cc:1035] T 37e4e8562d25473db32d95236b51ee8a P 50e8c97891d74b38b5c130d3bcbabcb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_10'
INSERT report
rows total: 2000
time total: 255.612 ms
time per row: 0.127806 ms
I20250627 13:52:39.338742 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49852:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250627 13:52:39.349078 971 tablet_service.cc:1468] Processing CreateTablet for tablet fc6f41fce2cd492eb1907ad29240d2f6 (DEFAULT_TABLE table=rebalance_test_table_11 [id=cfadf96fd2384254a8b165be87995a39]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:39.349184 1233 tablet_service.cc:1468] Processing CreateTablet for tablet fc6f41fce2cd492eb1907ad29240d2f6 (DEFAULT_TABLE table=rebalance_test_table_11 [id=cfadf96fd2384254a8b165be87995a39]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:39.349579 971 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc6f41fce2cd492eb1907ad29240d2f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:39.349620 1233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc6f41fce2cd492eb1907ad29240d2f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:39.349747 709 tablet_service.cc:1468] Processing CreateTablet for tablet d702d7deb99f4a2c9f3a534bb77e729e (DEFAULT_TABLE table=rebalance_test_table_11 [id=cfadf96fd2384254a8b165be87995a39]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:39.350135 709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d702d7deb99f4a2c9f3a534bb77e729e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:39.350075 710 tablet_service.cc:1468] Processing CreateTablet for tablet fc6f41fce2cd492eb1907ad29240d2f6 (DEFAULT_TABLE table=rebalance_test_table_11 [id=cfadf96fd2384254a8b165be87995a39]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:39.350396 710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc6f41fce2cd492eb1907ad29240d2f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:39.354197 1103 tablet_service.cc:1468] Processing CreateTablet for tablet 081c90892a7c4553a2f5d38f623c0914 (DEFAULT_TABLE table=rebalance_test_table_11 [id=cfadf96fd2384254a8b165be87995a39]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:39.354566 1103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 081c90892a7c4553a2f5d38f623c0914. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:39.358408 578 tablet_service.cc:1468] Processing CreateTablet for tablet 081c90892a7c4553a2f5d38f623c0914 (DEFAULT_TABLE table=rebalance_test_table_11 [id=cfadf96fd2384254a8b165be87995a39]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:39.358832 578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 081c90892a7c4553a2f5d38f623c0914. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:39.362478 840 tablet_service.cc:1468] Processing CreateTablet for tablet 081c90892a7c4553a2f5d38f623c0914 (DEFAULT_TABLE table=rebalance_test_table_11 [id=cfadf96fd2384254a8b165be87995a39]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:39.362870 840 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 081c90892a7c4553a2f5d38f623c0914. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:39.369091 972 tablet_service.cc:1468] Processing CreateTablet for tablet d702d7deb99f4a2c9f3a534bb77e729e (DEFAULT_TABLE table=rebalance_test_table_11 [id=cfadf96fd2384254a8b165be87995a39]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:39.369525 972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d702d7deb99f4a2c9f3a534bb77e729e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:39.369951 2349 tablet_bootstrap.cc:492] T fc6f41fce2cd492eb1907ad29240d2f6 P 50e8c97891d74b38b5c130d3bcbabcb6: Bootstrap starting.
I20250627 13:52:39.356186 1234 tablet_service.cc:1468] Processing CreateTablet for tablet d702d7deb99f4a2c9f3a534bb77e729e (DEFAULT_TABLE table=rebalance_test_table_11 [id=cfadf96fd2384254a8b165be87995a39]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:39.371021 1234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d702d7deb99f4a2c9f3a534bb77e729e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:39.371062 2352 tablet_bootstrap.cc:492] T 081c90892a7c4553a2f5d38f623c0914 P 957a8a8be3f542f48e8824e581fd0f89: Bootstrap starting.
I20250627 13:52:39.374264 2352 tablet_bootstrap.cc:654] T 081c90892a7c4553a2f5d38f623c0914 P 957a8a8be3f542f48e8824e581fd0f89: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:39.383528 2355 tablet_bootstrap.cc:492] T 081c90892a7c4553a2f5d38f623c0914 P 9f767210a91f44ae8b83b994131eee98: Bootstrap starting.
I20250627 13:52:39.384186 2351 tablet_bootstrap.cc:492] T 081c90892a7c4553a2f5d38f623c0914 P fa66786862da49ae826fb519c7d89b5e: Bootstrap starting.
I20250627 13:52:39.384415 2354 tablet_bootstrap.cc:492] T fc6f41fce2cd492eb1907ad29240d2f6 P 5c28ff2fe9424379819d478fdd093cfc: Bootstrap starting.
I20250627 13:52:39.385727 2355 tablet_bootstrap.cc:654] T 081c90892a7c4553a2f5d38f623c0914 P 9f767210a91f44ae8b83b994131eee98: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:39.386633 2351 tablet_bootstrap.cc:654] T 081c90892a7c4553a2f5d38f623c0914 P fa66786862da49ae826fb519c7d89b5e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:39.386808 2354 tablet_bootstrap.cc:654] T fc6f41fce2cd492eb1907ad29240d2f6 P 5c28ff2fe9424379819d478fdd093cfc: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:39.386888 2352 tablet_bootstrap.cc:492] T 081c90892a7c4553a2f5d38f623c0914 P 957a8a8be3f542f48e8824e581fd0f89: No bootstrap required, opened a new log
I20250627 13:52:39.387030 2352 ts_tablet_manager.cc:1397] T 081c90892a7c4553a2f5d38f623c0914 P 957a8a8be3f542f48e8824e581fd0f89: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.009s
I20250627 13:52:39.387881 2352 raft_consensus.cc:357] T 081c90892a7c4553a2f5d38f623c0914 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:39.388073 2352 raft_consensus.cc:383] T 081c90892a7c4553a2f5d38f623c0914 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:39.388128 2350 tablet_bootstrap.cc:492] T d702d7deb99f4a2c9f3a534bb77e729e P 75c4f59060974a3b823305b9dd339a7e: Bootstrap starting.
I20250627 13:52:39.388152 2352 raft_consensus.cc:738] T 081c90892a7c4553a2f5d38f623c0914 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 957a8a8be3f542f48e8824e581fd0f89, State: Initialized, Role: FOLLOWER
I20250627 13:52:39.388389 2352 consensus_queue.cc:260] T 081c90892a7c4553a2f5d38f623c0914 P 957a8a8be3f542f48e8824e581fd0f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:39.389812 2352 ts_tablet_manager.cc:1428] T 081c90892a7c4553a2f5d38f623c0914 P 957a8a8be3f542f48e8824e581fd0f89: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250627 13:52:39.390209 2350 tablet_bootstrap.cc:654] T d702d7deb99f4a2c9f3a534bb77e729e P 75c4f59060974a3b823305b9dd339a7e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:39.394755 2349 tablet_bootstrap.cc:654] T fc6f41fce2cd492eb1907ad29240d2f6 P 50e8c97891d74b38b5c130d3bcbabcb6: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:39.396093 2350 tablet_bootstrap.cc:492] T d702d7deb99f4a2c9f3a534bb77e729e P 75c4f59060974a3b823305b9dd339a7e: No bootstrap required, opened a new log
I20250627 13:52:39.396287 2350 ts_tablet_manager.cc:1397] T d702d7deb99f4a2c9f3a534bb77e729e P 75c4f59060974a3b823305b9dd339a7e: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20250627 13:52:39.396566 2354 tablet_bootstrap.cc:492] T fc6f41fce2cd492eb1907ad29240d2f6 P 5c28ff2fe9424379819d478fdd093cfc: No bootstrap required, opened a new log
I20250627 13:52:39.396799 2354 ts_tablet_manager.cc:1397] T fc6f41fce2cd492eb1907ad29240d2f6 P 5c28ff2fe9424379819d478fdd093cfc: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.005s
I20250627 13:52:39.397109 2350 raft_consensus.cc:357] T d702d7deb99f4a2c9f3a534bb77e729e P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:39.397378 2350 raft_consensus.cc:383] T d702d7deb99f4a2c9f3a534bb77e729e P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:39.397540 2350 raft_consensus.cc:738] T d702d7deb99f4a2c9f3a534bb77e729e P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c4f59060974a3b823305b9dd339a7e, State: Initialized, Role: FOLLOWER
I20250627 13:52:39.397853 2350 consensus_queue.cc:260] T d702d7deb99f4a2c9f3a534bb77e729e P 75c4f59060974a3b823305b9dd339a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:39.397809 2354 raft_consensus.cc:357] T fc6f41fce2cd492eb1907ad29240d2f6 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:39.398037 2354 raft_consensus.cc:383] T fc6f41fce2cd492eb1907ad29240d2f6 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:39.398196 2354 raft_consensus.cc:738] T fc6f41fce2cd492eb1907ad29240d2f6 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c28ff2fe9424379819d478fdd093cfc, State: Initialized, Role: FOLLOWER
I20250627 13:52:39.398486 2351 tablet_bootstrap.cc:492] T 081c90892a7c4553a2f5d38f623c0914 P fa66786862da49ae826fb519c7d89b5e: No bootstrap required, opened a new log
I20250627 13:52:39.398478 2354 consensus_queue.cc:260] T fc6f41fce2cd492eb1907ad29240d2f6 P 5c28ff2fe9424379819d478fdd093cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:39.398593 2350 ts_tablet_manager.cc:1428] T d702d7deb99f4a2c9f3a534bb77e729e P 75c4f59060974a3b823305b9dd339a7e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:52:39.398648 2351 ts_tablet_manager.cc:1397] T 081c90892a7c4553a2f5d38f623c0914 P fa66786862da49ae826fb519c7d89b5e: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.005s
I20250627 13:52:39.398873 2350 tablet_bootstrap.cc:492] T fc6f41fce2cd492eb1907ad29240d2f6 P 75c4f59060974a3b823305b9dd339a7e: Bootstrap starting.
I20250627 13:52:39.399351 2354 ts_tablet_manager.cc:1428] T fc6f41fce2cd492eb1907ad29240d2f6 P 5c28ff2fe9424379819d478fdd093cfc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:52:39.399441 2351 raft_consensus.cc:357] T 081c90892a7c4553a2f5d38f623c0914 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:39.399616 2354 tablet_bootstrap.cc:492] T d702d7deb99f4a2c9f3a534bb77e729e P 5c28ff2fe9424379819d478fdd093cfc: Bootstrap starting.
I20250627 13:52:39.399624 2351 raft_consensus.cc:383] T 081c90892a7c4553a2f5d38f623c0914 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:39.399717 2351 raft_consensus.cc:738] T 081c90892a7c4553a2f5d38f623c0914 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa66786862da49ae826fb519c7d89b5e, State: Initialized, Role: FOLLOWER
I20250627 13:52:39.399933 2351 consensus_queue.cc:260] T 081c90892a7c4553a2f5d38f623c0914 P fa66786862da49ae826fb519c7d89b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:39.400774 2351 ts_tablet_manager.cc:1428] T 081c90892a7c4553a2f5d38f623c0914 P fa66786862da49ae826fb519c7d89b5e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250627 13:52:39.401453 2354 tablet_bootstrap.cc:654] T d702d7deb99f4a2c9f3a534bb77e729e P 5c28ff2fe9424379819d478fdd093cfc: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:39.402679 2350 tablet_bootstrap.cc:654] T fc6f41fce2cd492eb1907ad29240d2f6 P 75c4f59060974a3b823305b9dd339a7e: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:39.406737 2355 tablet_bootstrap.cc:492] T 081c90892a7c4553a2f5d38f623c0914 P 9f767210a91f44ae8b83b994131eee98: No bootstrap required, opened a new log
I20250627 13:52:39.406953 2355 ts_tablet_manager.cc:1397] T 081c90892a7c4553a2f5d38f623c0914 P 9f767210a91f44ae8b83b994131eee98: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.005s
I20250627 13:52:39.408123 2355 raft_consensus.cc:357] T 081c90892a7c4553a2f5d38f623c0914 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:39.408404 2355 raft_consensus.cc:383] T 081c90892a7c4553a2f5d38f623c0914 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:39.408557 2355 raft_consensus.cc:738] T 081c90892a7c4553a2f5d38f623c0914 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f767210a91f44ae8b83b994131eee98, State: Initialized, Role: FOLLOWER
I20250627 13:52:39.408890 2355 consensus_queue.cc:260] T 081c90892a7c4553a2f5d38f623c0914 P 9f767210a91f44ae8b83b994131eee98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:39.409147 2350 tablet_bootstrap.cc:492] T fc6f41fce2cd492eb1907ad29240d2f6 P 75c4f59060974a3b823305b9dd339a7e: No bootstrap required, opened a new log
I20250627 13:52:39.409336 2350 ts_tablet_manager.cc:1397] T fc6f41fce2cd492eb1907ad29240d2f6 P 75c4f59060974a3b823305b9dd339a7e: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20250627 13:52:39.409773 2355 ts_tablet_manager.cc:1428] T 081c90892a7c4553a2f5d38f623c0914 P 9f767210a91f44ae8b83b994131eee98: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250627 13:52:39.410183 2350 raft_consensus.cc:357] T fc6f41fce2cd492eb1907ad29240d2f6 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:39.410457 2350 raft_consensus.cc:383] T fc6f41fce2cd492eb1907ad29240d2f6 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:39.410581 2350 raft_consensus.cc:738] T fc6f41fce2cd492eb1907ad29240d2f6 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c4f59060974a3b823305b9dd339a7e, State: Initialized, Role: FOLLOWER
I20250627 13:52:39.410856 2350 consensus_queue.cc:260] T fc6f41fce2cd492eb1907ad29240d2f6 P 75c4f59060974a3b823305b9dd339a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:39.411357 2350 ts_tablet_manager.cc:1428] T fc6f41fce2cd492eb1907ad29240d2f6 P 75c4f59060974a3b823305b9dd339a7e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:52:39.412521 2354 tablet_bootstrap.cc:492] T d702d7deb99f4a2c9f3a534bb77e729e P 5c28ff2fe9424379819d478fdd093cfc: No bootstrap required, opened a new log
I20250627 13:52:39.412698 2354 ts_tablet_manager.cc:1397] T d702d7deb99f4a2c9f3a534bb77e729e P 5c28ff2fe9424379819d478fdd093cfc: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20250627 13:52:39.413491 2354 raft_consensus.cc:357] T d702d7deb99f4a2c9f3a534bb77e729e P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:39.413695 2354 raft_consensus.cc:383] T d702d7deb99f4a2c9f3a534bb77e729e P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:39.413779 2354 raft_consensus.cc:738] T d702d7deb99f4a2c9f3a534bb77e729e P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c28ff2fe9424379819d478fdd093cfc, State: Initialized, Role: FOLLOWER
I20250627 13:52:39.414021 2354 consensus_queue.cc:260] T d702d7deb99f4a2c9f3a534bb77e729e P 5c28ff2fe9424379819d478fdd093cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:39.414635 2354 ts_tablet_manager.cc:1428] T d702d7deb99f4a2c9f3a534bb77e729e P 5c28ff2fe9424379819d478fdd093cfc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:52:39.415733 1509 raft_consensus.cc:491] T d702d7deb99f4a2c9f3a534bb77e729e P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:39.415889 1509 raft_consensus.cc:513] T d702d7deb99f4a2c9f3a534bb77e729e P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:39.416553 2349 tablet_bootstrap.cc:492] T fc6f41fce2cd492eb1907ad29240d2f6 P 50e8c97891d74b38b5c130d3bcbabcb6: No bootstrap required, opened a new log
I20250627 13:52:39.416685 2349 ts_tablet_manager.cc:1397] T fc6f41fce2cd492eb1907ad29240d2f6 P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.002s
I20250627 13:52:39.416697 1509 leader_election.cc:290] T d702d7deb99f4a2c9f3a534bb77e729e P 75c4f59060974a3b823305b9dd339a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70:39887), 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68:34289)
I20250627 13:52:39.417043 1253 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d702d7deb99f4a2c9f3a534bb77e729e" candidate_uuid: "75c4f59060974a3b823305b9dd339a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c28ff2fe9424379819d478fdd093cfc" is_pre_election: true
I20250627 13:52:39.417240 1253 raft_consensus.cc:2466] T d702d7deb99f4a2c9f3a534bb77e729e P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75c4f59060974a3b823305b9dd339a7e in term 0.
I20250627 13:52:39.417250 992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d702d7deb99f4a2c9f3a534bb77e729e" candidate_uuid: "75c4f59060974a3b823305b9dd339a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" is_pre_election: true
I20250627 13:52:39.417441 2349 raft_consensus.cc:357] T fc6f41fce2cd492eb1907ad29240d2f6 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:39.417721 2349 raft_consensus.cc:383] T fc6f41fce2cd492eb1907ad29240d2f6 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:39.417742 664 leader_election.cc:304] T d702d7deb99f4a2c9f3a534bb77e729e P 75c4f59060974a3b823305b9dd339a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c28ff2fe9424379819d478fdd093cfc, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:39.417863 2349 raft_consensus.cc:738] T fc6f41fce2cd492eb1907ad29240d2f6 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e8c97891d74b38b5c130d3bcbabcb6, State: Initialized, Role: FOLLOWER
I20250627 13:52:39.418082 2349 consensus_queue.cc:260] T fc6f41fce2cd492eb1907ad29240d2f6 P 50e8c97891d74b38b5c130d3bcbabcb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:39.418227 1509 raft_consensus.cc:2802] T d702d7deb99f4a2c9f3a534bb77e729e P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:39.418339 1509 raft_consensus.cc:491] T d702d7deb99f4a2c9f3a534bb77e729e P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250627 13:52:39.418315 665 leader_election.cc:343] T d702d7deb99f4a2c9f3a534bb77e729e P 75c4f59060974a3b823305b9dd339a7e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68:34289): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:52:39.418438 1509 raft_consensus.cc:3058] T d702d7deb99f4a2c9f3a534bb77e729e P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:39.418717 2349 ts_tablet_manager.cc:1428] T fc6f41fce2cd492eb1907ad29240d2f6 P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250627 13:52:39.418942 2349 tablet_bootstrap.cc:492] T d702d7deb99f4a2c9f3a534bb77e729e P 50e8c97891d74b38b5c130d3bcbabcb6: Bootstrap starting.
I20250627 13:52:39.420625 2349 tablet_bootstrap.cc:654] T d702d7deb99f4a2c9f3a534bb77e729e P 50e8c97891d74b38b5c130d3bcbabcb6: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:39.421841 1509 raft_consensus.cc:513] T d702d7deb99f4a2c9f3a534bb77e729e P 75c4f59060974a3b823305b9dd339a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:39.422190 1790 raft_consensus.cc:491] T 081c90892a7c4553a2f5d38f623c0914 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:39.422346 1790 raft_consensus.cc:513] T 081c90892a7c4553a2f5d38f623c0914 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:39.422767 1253 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d702d7deb99f4a2c9f3a534bb77e729e" candidate_uuid: "75c4f59060974a3b823305b9dd339a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c28ff2fe9424379819d478fdd093cfc"
I20250627 13:52:39.422956 1253 raft_consensus.cc:3058] T d702d7deb99f4a2c9f3a534bb77e729e P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:39.423266 1790 leader_election.cc:290] T 081c90892a7c4553a2f5d38f623c0914 P fa66786862da49ae826fb519c7d89b5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023), 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69:42325)
I20250627 13:52:39.423705 2273 raft_consensus.cc:491] T 081c90892a7c4553a2f5d38f623c0914 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:39.423832 2349 tablet_bootstrap.cc:492] T d702d7deb99f4a2c9f3a534bb77e729e P 50e8c97891d74b38b5c130d3bcbabcb6: No bootstrap required, opened a new log
I20250627 13:52:39.423931 2349 ts_tablet_manager.cc:1397] T d702d7deb99f4a2c9f3a534bb77e729e P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20250627 13:52:39.423851 2273 raft_consensus.cc:513] T 081c90892a7c4553a2f5d38f623c0914 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:39.424661 2273 leader_election.cc:290] T 081c90892a7c4553a2f5d38f623c0914 P 9f767210a91f44ae8b83b994131eee98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747), 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69:42325)
I20250627 13:52:39.424883 2349 raft_consensus.cc:357] T d702d7deb99f4a2c9f3a534bb77e729e P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:39.425057 2349 raft_consensus.cc:383] T d702d7deb99f4a2c9f3a534bb77e729e P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:39.425136 2349 raft_consensus.cc:738] T d702d7deb99f4a2c9f3a534bb77e729e P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e8c97891d74b38b5c130d3bcbabcb6, State: Initialized, Role: FOLLOWER
I20250627 13:52:39.425231 860 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "081c90892a7c4553a2f5d38f623c0914" candidate_uuid: "9f767210a91f44ae8b83b994131eee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e" is_pre_election: true
I20250627 13:52:39.425390 860 raft_consensus.cc:2466] T 081c90892a7c4553a2f5d38f623c0914 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f767210a91f44ae8b83b994131eee98 in term 0.
I20250627 13:52:39.425489 2349 consensus_queue.cc:260] T d702d7deb99f4a2c9f3a534bb77e729e P 50e8c97891d74b38b5c130d3bcbabcb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:39.425689 1253 raft_consensus.cc:2466] T d702d7deb99f4a2c9f3a534bb77e729e P 5c28ff2fe9424379819d478fdd093cfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75c4f59060974a3b823305b9dd339a7e in term 1.
I20250627 13:52:39.425884 2349 ts_tablet_manager.cc:1428] T d702d7deb99f4a2c9f3a534bb77e729e P 50e8c97891d74b38b5c130d3bcbabcb6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:52:39.426062 598 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "081c90892a7c4553a2f5d38f623c0914" candidate_uuid: "fa66786862da49ae826fb519c7d89b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98" is_pre_election: true
I20250627 13:52:39.426126 1509 leader_election.cc:290] T d702d7deb99f4a2c9f3a534bb77e729e P 75c4f59060974a3b823305b9dd339a7e [CANDIDATE]: Term 1 election: Requested vote from peers 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70:39887), 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68:34289)
I20250627 13:52:39.426146 664 leader_election.cc:304] T d702d7deb99f4a2c9f3a534bb77e729e P 75c4f59060974a3b823305b9dd339a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c28ff2fe9424379819d478fdd093cfc, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:39.426208 1123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "081c90892a7c4553a2f5d38f623c0914" candidate_uuid: "9f767210a91f44ae8b83b994131eee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957a8a8be3f542f48e8824e581fd0f89" is_pre_election: true
I20250627 13:52:39.426249 598 raft_consensus.cc:2466] T 081c90892a7c4553a2f5d38f623c0914 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa66786862da49ae826fb519c7d89b5e in term 0.
I20250627 13:52:39.426398 1123 raft_consensus.cc:2466] T 081c90892a7c4553a2f5d38f623c0914 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f767210a91f44ae8b83b994131eee98 in term 0.
I20250627 13:52:39.426784 534 leader_election.cc:304] T 081c90892a7c4553a2f5d38f623c0914 P 9f767210a91f44ae8b83b994131eee98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, 9f767210a91f44ae8b83b994131eee98; no voters:
I20250627 13:52:39.426842 992 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d702d7deb99f4a2c9f3a534bb77e729e" candidate_uuid: "75c4f59060974a3b823305b9dd339a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e8c97891d74b38b5c130d3bcbabcb6"
I20250627 13:52:39.426910 1123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "081c90892a7c4553a2f5d38f623c0914" candidate_uuid: "fa66786862da49ae826fb519c7d89b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957a8a8be3f542f48e8824e581fd0f89" is_pre_election: true
I20250627 13:52:39.427003 992 raft_consensus.cc:3058] T d702d7deb99f4a2c9f3a534bb77e729e P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:39.427076 1123 raft_consensus.cc:2466] T 081c90892a7c4553a2f5d38f623c0914 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa66786862da49ae826fb519c7d89b5e in term 0.
I20250627 13:52:39.427145 794 leader_election.cc:304] T 081c90892a7c4553a2f5d38f623c0914 P fa66786862da49ae826fb519c7d89b5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f767210a91f44ae8b83b994131eee98, fa66786862da49ae826fb519c7d89b5e; no voters:
I20250627 13:52:39.427474 2273 raft_consensus.cc:2802] T 081c90892a7c4553a2f5d38f623c0914 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:39.427593 2273 raft_consensus.cc:491] T 081c90892a7c4553a2f5d38f623c0914 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:39.427680 2273 raft_consensus.cc:3058] T 081c90892a7c4553a2f5d38f623c0914 P 9f767210a91f44ae8b83b994131eee98 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:39.427794 1509 raft_consensus.cc:2802] T d702d7deb99f4a2c9f3a534bb77e729e P 75c4f59060974a3b823305b9dd339a7e [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:39.428187 1509 raft_consensus.cc:695] T d702d7deb99f4a2c9f3a534bb77e729e P 75c4f59060974a3b823305b9dd339a7e [term 1 LEADER]: Becoming Leader. State: Replica: 75c4f59060974a3b823305b9dd339a7e, State: Running, Role: LEADER
I20250627 13:52:39.428472 1509 consensus_queue.cc:237] T d702d7deb99f4a2c9f3a534bb77e729e P 75c4f59060974a3b823305b9dd339a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } }
I20250627 13:52:39.429795 992 raft_consensus.cc:2466] T d702d7deb99f4a2c9f3a534bb77e729e P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75c4f59060974a3b823305b9dd339a7e in term 1.
I20250627 13:52:39.431073 1790 raft_consensus.cc:2802] T 081c90892a7c4553a2f5d38f623c0914 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:39.431205 1790 raft_consensus.cc:491] T 081c90892a7c4553a2f5d38f623c0914 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:39.430339 2273 raft_consensus.cc:513] T 081c90892a7c4553a2f5d38f623c0914 P 9f767210a91f44ae8b83b994131eee98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:39.431298 1790 raft_consensus.cc:3058] T 081c90892a7c4553a2f5d38f623c0914 P fa66786862da49ae826fb519c7d89b5e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:39.432052 1123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "081c90892a7c4553a2f5d38f623c0914" candidate_uuid: "9f767210a91f44ae8b83b994131eee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957a8a8be3f542f48e8824e581fd0f89"
I20250627 13:52:39.432034 861 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "081c90892a7c4553a2f5d38f623c0914" candidate_uuid: "9f767210a91f44ae8b83b994131eee98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa66786862da49ae826fb519c7d89b5e"
I20250627 13:52:39.431885 449 catalog_manager.cc:5582] T d702d7deb99f4a2c9f3a534bb77e729e P 75c4f59060974a3b823305b9dd339a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to 75c4f59060974a3b823305b9dd339a7e (127.0.103.66). New cstate: current_term: 1 leader_uuid: "75c4f59060974a3b823305b9dd339a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } health_report { overall_health: UNKNOWN } } }
I20250627 13:52:39.432207 1123 raft_consensus.cc:3058] T 081c90892a7c4553a2f5d38f623c0914 P 957a8a8be3f542f48e8824e581fd0f89 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:39.434741 1123 raft_consensus.cc:2466] T 081c90892a7c4553a2f5d38f623c0914 P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f767210a91f44ae8b83b994131eee98 in term 1.
I20250627 13:52:39.435117 1790 raft_consensus.cc:513] T 081c90892a7c4553a2f5d38f623c0914 P fa66786862da49ae826fb519c7d89b5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:39.435186 534 leader_election.cc:304] T 081c90892a7c4553a2f5d38f623c0914 P 9f767210a91f44ae8b83b994131eee98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 957a8a8be3f542f48e8824e581fd0f89, 9f767210a91f44ae8b83b994131eee98; no voters:
I20250627 13:52:39.435487 2287 raft_consensus.cc:2802] T 081c90892a7c4553a2f5d38f623c0914 P 9f767210a91f44ae8b83b994131eee98 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:39.435628 861 raft_consensus.cc:2391] T 081c90892a7c4553a2f5d38f623c0914 P fa66786862da49ae826fb519c7d89b5e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f767210a91f44ae8b83b994131eee98 in current term 1: Already voted for candidate fa66786862da49ae826fb519c7d89b5e in this term.
I20250627 13:52:39.435665 2287 raft_consensus.cc:695] T 081c90892a7c4553a2f5d38f623c0914 P 9f767210a91f44ae8b83b994131eee98 [term 1 LEADER]: Becoming Leader. State: Replica: 9f767210a91f44ae8b83b994131eee98, State: Running, Role: LEADER
I20250627 13:52:39.435957 2287 consensus_queue.cc:237] T 081c90892a7c4553a2f5d38f623c0914 P 9f767210a91f44ae8b83b994131eee98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } }
I20250627 13:52:39.436770 1790 leader_election.cc:290] T 081c90892a7c4553a2f5d38f623c0914 P fa66786862da49ae826fb519c7d89b5e [CANDIDATE]: Term 1 election: Requested vote from peers 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023), 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69:42325)
I20250627 13:52:39.437130 598 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "081c90892a7c4553a2f5d38f623c0914" candidate_uuid: "fa66786862da49ae826fb519c7d89b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f767210a91f44ae8b83b994131eee98"
I20250627 13:52:39.437321 1123 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "081c90892a7c4553a2f5d38f623c0914" candidate_uuid: "fa66786862da49ae826fb519c7d89b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "957a8a8be3f542f48e8824e581fd0f89"
I20250627 13:52:39.437558 1123 raft_consensus.cc:2391] T 081c90892a7c4553a2f5d38f623c0914 P 957a8a8be3f542f48e8824e581fd0f89 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fa66786862da49ae826fb519c7d89b5e in current term 1: Already voted for candidate 9f767210a91f44ae8b83b994131eee98 in this term.
I20250627 13:52:39.437999 2273 leader_election.cc:290] T 081c90892a7c4553a2f5d38f623c0914 P 9f767210a91f44ae8b83b994131eee98 [CANDIDATE]: Term 1 election: Requested vote from peers fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747), 957a8a8be3f542f48e8824e581fd0f89 (127.0.103.69:42325)
I20250627 13:52:39.438000 797 leader_election.cc:304] T 081c90892a7c4553a2f5d38f623c0914 P fa66786862da49ae826fb519c7d89b5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa66786862da49ae826fb519c7d89b5e; no voters: 957a8a8be3f542f48e8824e581fd0f89, 9f767210a91f44ae8b83b994131eee98
I20250627 13:52:39.438254 1790 raft_consensus.cc:2747] T 081c90892a7c4553a2f5d38f623c0914 P fa66786862da49ae826fb519c7d89b5e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250627 13:52:39.439208 451 catalog_manager.cc:5582] T 081c90892a7c4553a2f5d38f623c0914 P 9f767210a91f44ae8b83b994131eee98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f767210a91f44ae8b83b994131eee98 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "9f767210a91f44ae8b83b994131eee98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f767210a91f44ae8b83b994131eee98" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 43023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 } health_report { overall_health: UNKNOWN } } }
I20250627 13:52:39.516640 2277 raft_consensus.cc:491] T fc6f41fce2cd492eb1907ad29240d2f6 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:39.516885 2277 raft_consensus.cc:513] T fc6f41fce2cd492eb1907ad29240d2f6 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:39.517825 2277 leader_election.cc:290] T fc6f41fce2cd492eb1907ad29240d2f6 P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70:39887), 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499)
I20250627 13:52:39.518299 1253 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6f41fce2cd492eb1907ad29240d2f6" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c28ff2fe9424379819d478fdd093cfc" is_pre_election: true
I20250627 13:52:39.518339 728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6f41fce2cd492eb1907ad29240d2f6" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c4f59060974a3b823305b9dd339a7e" is_pre_election: true
I20250627 13:52:39.518496 1253 raft_consensus.cc:2466] T fc6f41fce2cd492eb1907ad29240d2f6 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 0.
I20250627 13:52:39.518566 728 raft_consensus.cc:2466] T fc6f41fce2cd492eb1907ad29240d2f6 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 0.
I20250627 13:52:39.518963 926 leader_election.cc:304] T fc6f41fce2cd492eb1907ad29240d2f6 P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50e8c97891d74b38b5c130d3bcbabcb6, 5c28ff2fe9424379819d478fdd093cfc; no voters:
I20250627 13:52:39.519227 2277 raft_consensus.cc:2802] T fc6f41fce2cd492eb1907ad29240d2f6 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:39.519338 2277 raft_consensus.cc:491] T fc6f41fce2cd492eb1907ad29240d2f6 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:39.519466 2277 raft_consensus.cc:3058] T fc6f41fce2cd492eb1907ad29240d2f6 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:39.521633 2277 raft_consensus.cc:513] T fc6f41fce2cd492eb1907ad29240d2f6 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:39.522212 2277 leader_election.cc:290] T fc6f41fce2cd492eb1907ad29240d2f6 P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 election: Requested vote from peers 5c28ff2fe9424379819d478fdd093cfc (127.0.103.70:39887), 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499)
I20250627 13:52:39.522553 1253 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6f41fce2cd492eb1907ad29240d2f6" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c28ff2fe9424379819d478fdd093cfc"
I20250627 13:52:39.522785 1253 raft_consensus.cc:3058] T fc6f41fce2cd492eb1907ad29240d2f6 P 5c28ff2fe9424379819d478fdd093cfc [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:39.522774 728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fc6f41fce2cd492eb1907ad29240d2f6" candidate_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75c4f59060974a3b823305b9dd339a7e"
I20250627 13:52:39.522964 728 raft_consensus.cc:3058] T fc6f41fce2cd492eb1907ad29240d2f6 P 75c4f59060974a3b823305b9dd339a7e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:39.525727 728 raft_consensus.cc:2466] T fc6f41fce2cd492eb1907ad29240d2f6 P 75c4f59060974a3b823305b9dd339a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 1.
I20250627 13:52:39.525796 1253 raft_consensus.cc:2466] T fc6f41fce2cd492eb1907ad29240d2f6 P 5c28ff2fe9424379819d478fdd093cfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50e8c97891d74b38b5c130d3bcbabcb6 in term 1.
I20250627 13:52:39.526135 928 leader_election.cc:304] T fc6f41fce2cd492eb1907ad29240d2f6 P 50e8c97891d74b38b5c130d3bcbabcb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50e8c97891d74b38b5c130d3bcbabcb6, 75c4f59060974a3b823305b9dd339a7e; no voters:
I20250627 13:52:39.526453 2277 raft_consensus.cc:2802] T fc6f41fce2cd492eb1907ad29240d2f6 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:39.526763 2277 raft_consensus.cc:695] T fc6f41fce2cd492eb1907ad29240d2f6 P 50e8c97891d74b38b5c130d3bcbabcb6 [term 1 LEADER]: Becoming Leader. State: Replica: 50e8c97891d74b38b5c130d3bcbabcb6, State: Running, Role: LEADER
I20250627 13:52:39.527024 2277 consensus_queue.cc:237] T fc6f41fce2cd492eb1907ad29240d2f6 P 50e8c97891d74b38b5c130d3bcbabcb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } }
I20250627 13:52:39.529770 451 catalog_manager.cc:5582] T fc6f41fce2cd492eb1907ad29240d2f6 P 50e8c97891d74b38b5c130d3bcbabcb6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68). New cstate: current_term: 1 leader_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 } health_report { overall_health: UNKNOWN } } }
I20250627 13:52:39.864601 2273 consensus_queue.cc:1035] T 081c90892a7c4553a2f5d38f623c0914 P 9f767210a91f44ae8b83b994131eee98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "957a8a8be3f542f48e8824e581fd0f89" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:39.872356 2273 consensus_queue.cc:1035] T 081c90892a7c4553a2f5d38f623c0914 P 9f767210a91f44ae8b83b994131eee98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa66786862da49ae826fb519c7d89b5e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250627 13:52:39.897836 2368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:39.898187 2368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:39.906304 2368 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20250627 13:52:39.943276 1509 consensus_queue.cc:1035] T d702d7deb99f4a2c9f3a534bb77e729e P 75c4f59060974a3b823305b9dd339a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:39.948490 2391 consensus_queue.cc:1035] T d702d7deb99f4a2c9f3a534bb77e729e P 75c4f59060974a3b823305b9dd339a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "50e8c97891d74b38b5c130d3bcbabcb6" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:39.949411 2277 consensus_queue.cc:1035] T fc6f41fce2cd492eb1907ad29240d2f6 P 50e8c97891d74b38b5c130d3bcbabcb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c28ff2fe9424379819d478fdd093cfc" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 39887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:39.969568 2306 consensus_queue.cc:1035] T fc6f41fce2cd492eb1907ad29240d2f6 P 50e8c97891d74b38b5c130d3bcbabcb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75c4f59060974a3b823305b9dd339a7e" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 276.578 ms
time per row: 0.138289 ms
I20250627 13:52:40.888895 972 tablet_service.cc:1430] Tablet server has 8 leaders and 0 scanners
I20250627 13:52:40.907049 709 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:52:40.914258 1233 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:52:40.930052 1103 tablet_service.cc:1430] Tablet server has 9 leaders and 0 scanners
I20250627 13:52:40.937704 840 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:52:40.967104 578 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:52:41.176030 1233 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:52:41.179119 972 tablet_service.cc:1430] Tablet server has 8 leaders and 0 scanners
I20250627 13:52:41.182636 709 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:52:41.199169 840 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:52:41.206346 578 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:52:41.212162 1103 tablet_service.cc:1430] Tablet server has 9 leaders and 0 scanners
I20250627 13:52:41.494020 972 tablet_service.cc:1430] Tablet server has 8 leaders and 0 scanners
I20250627 13:52:41.502120 1233 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:52:41.517098 709 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:52:41.536358 578 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:52:41.541949 1103 tablet_service.cc:1430] Tablet server has 9 leaders and 0 scanners
I20250627 13:52:41.549490 840 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:52:41.770401 972 tablet_service.cc:1430] Tablet server has 8 leaders and 0 scanners
I20250627 13:52:41.779749 709 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:52:41.788882 1103 tablet_service.cc:1430] Tablet server has 9 leaders and 0 scanners
I20250627 13:52:41.789332 1233 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:52:41.795120 578 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:52:41.795177 840 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:52:42.047600 972 tablet_service.cc:1430] Tablet server has 8 leaders and 0 scanners
I20250627 13:52:42.050518 1103 tablet_service.cc:1430] Tablet server has 9 leaders and 0 scanners
I20250627 13:52:42.051010 709 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:52:42.056378 1233 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:52:42.077464 578 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:52:42.079706 840 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
W20250627 13:52:42.141356 928 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.66:36499, user_credentials={real_user=slave}} blocked reactor thread for 77195.3us
I20250627 13:52:42.355314 972 tablet_service.cc:1430] Tablet server has 8 leaders and 0 scanners
I20250627 13:52:42.356204 1233 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:52:42.380060 578 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:52:42.390121 840 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:52:42.410205 709 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:52:42.424158 1103 tablet_service.cc:1430] Tablet server has 9 leaders and 0 scanners
I20250627 13:52:42.621217 972 tablet_service.cc:1430] Tablet server has 8 leaders and 0 scanners
I20250627 13:52:42.633289 1233 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:52:42.643306 709 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:52:42.647385 578 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:52:42.650841 1103 tablet_service.cc:1430] Tablet server has 9 leaders and 0 scanners
I20250627 13:52:42.663149 840 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:52:42.903141 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 516
I20250627 13:52:42.910017 638 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:52:43.045390 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 516
W20250627 13:52:43.070839 1056 connection.cc:537] client connection to 127.0.103.65:43023 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250627 13:52:43.071151 1056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250627 13:52:43.071280 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 647
W20250627 13:52:43.073179 1056 consensus_peers.cc:487] T a4be1509d80e400c9a1cd0686f38e47e P 957a8a8be3f542f48e8824e581fd0f89 -> Peer 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023): Couldn't send request to peer 9f767210a91f44ae8b83b994131eee98. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:43023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:52:43.073326 1056 consensus_peers.cc:487] T 46cbed417f5a4f2ebded566383ee3440 P 957a8a8be3f542f48e8824e581fd0f89 -> Peer 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023): Couldn't send request to peer 9f767210a91f44ae8b83b994131eee98. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:43023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250627 13:52:43.077889 769 generic_service.cc:196] Checking for leaks (request via RPC)
W20250627 13:52:43.149119 1056 consensus_peers.cc:487] T 574d86438dc240aca8fd3791c70d19b3 P 957a8a8be3f542f48e8824e581fd0f89 -> Peer 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023): Couldn't send request to peer 9f767210a91f44ae8b83b994131eee98. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:43023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:52:43.170147 1056 consensus_peers.cc:487] T f296de1facde47308828eb4ed2283079 P 957a8a8be3f542f48e8824e581fd0f89 -> Peer 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023): Couldn't send request to peer 9f767210a91f44ae8b83b994131eee98. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:43023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:52:43.180773 1056 consensus_peers.cc:487] T 012c8298a5d741909d6838470e8c05ba P 957a8a8be3f542f48e8824e581fd0f89 -> Peer 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023): Couldn't send request to peer 9f767210a91f44ae8b83b994131eee98. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:43023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:52:43.221258 794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.65:43023: connect: Connection refused (error 111)
I20250627 13:52:43.221664 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 647
W20250627 13:52:43.224527 794 consensus_peers.cc:487] T e14c8045e0374a649bf07ae94528f329 P fa66786862da49ae826fb519c7d89b5e -> Peer 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023): Couldn't send request to peer 9f767210a91f44ae8b83b994131eee98. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:43023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:52:43.245338 927 connection.cc:537] server connection from 127.0.103.66:50035 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250627 13:52:43.245970 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 778
I20250627 13:52:43.251755 900 generic_service.cc:196] Checking for leaks (request via RPC)
W20250627 13:52:43.253196 928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.66:36499: connect: Connection refused (error 111)
W20250627 13:52:43.254643 928 consensus_peers.cc:487] T 148debd8ad384f51b509200b90ad4fdc P 50e8c97891d74b38b5c130d3bcbabcb6 -> Peer 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499): Couldn't send request to peer 75c4f59060974a3b823305b9dd339a7e. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.66:36499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:52:43.254786 928 consensus_peers.cc:487] T f55615149da94dd1be6f5f09c4a2f026 P 50e8c97891d74b38b5c130d3bcbabcb6 -> Peer 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499): Couldn't send request to peer 75c4f59060974a3b823305b9dd339a7e. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.66:36499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:52:43.261495 928 consensus_peers.cc:487] T fc6f41fce2cd492eb1907ad29240d2f6 P 50e8c97891d74b38b5c130d3bcbabcb6 -> Peer 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499): Couldn't send request to peer 75c4f59060974a3b823305b9dd339a7e. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.66:36499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:52:43.262562 1190 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.66:36499: connect: Connection refused (error 111)
W20250627 13:52:43.267066 1190 consensus_peers.cc:487] T d146d21a389944a0b47e89e85a6fe920 P 5c28ff2fe9424379819d478fdd093cfc -> Peer 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499): Couldn't send request to peer 75c4f59060974a3b823305b9dd339a7e. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.66:36499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:52:43.280903 928 consensus_peers.cc:487] T f3721cf7c3a54f3e9e9cc88fc7ce7691 P 50e8c97891d74b38b5c130d3bcbabcb6 -> Peer 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499): Couldn't send request to peer 75c4f59060974a3b823305b9dd339a7e. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.66:36499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:52:43.285702 928 consensus_peers.cc:487] T fc3d1ee303e64241930642d9f13a3fd0 P 50e8c97891d74b38b5c130d3bcbabcb6 -> Peer 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499): Couldn't send request to peer 75c4f59060974a3b823305b9dd339a7e. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.66:36499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:52:43.321789 1056 consensus_peers.cc:487] T 0118ccc5093e4971a48f0270c2224959 P 957a8a8be3f542f48e8824e581fd0f89 -> Peer 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023): Couldn't send request to peer 9f767210a91f44ae8b83b994131eee98. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:43023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:52:43.356555 1056 consensus_peers.cc:487] T 76a58b66489c4dd68d5c037aaffa9835 P 957a8a8be3f542f48e8824e581fd0f89 -> Peer 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023): Couldn't send request to peer 9f767210a91f44ae8b83b994131eee98. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:43023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:52:43.373330 1056 consensus_peers.cc:487] T f3a32a4bc6ad45ebacea4f92ebabeb95 P 957a8a8be3f542f48e8824e581fd0f89 -> Peer 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023): Couldn't send request to peer 9f767210a91f44ae8b83b994131eee98. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:43023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:52:43.390542 928 consensus_peers.cc:487] T b77a6923c0494a5ea8471b899acff4df P 50e8c97891d74b38b5c130d3bcbabcb6 -> Peer 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499): Couldn't send request to peer 75c4f59060974a3b823305b9dd339a7e. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.66:36499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250627 13:52:43.392117 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 778
I20250627 13:52:43.416292 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 909
I20250627 13:52:43.422143 1031 generic_service.cc:196] Checking for leaks (request via RPC)
W20250627 13:52:43.433010 1058 consensus_peers.cc:487] T 46cbed417f5a4f2ebded566383ee3440 P 957a8a8be3f542f48e8824e581fd0f89 -> Peer fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747): Couldn't send request to peer fa66786862da49ae826fb519c7d89b5e. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.67:40747: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:52:43.447222 1058 consensus_peers.cc:487] T 574d86438dc240aca8fd3791c70d19b3 P 957a8a8be3f542f48e8824e581fd0f89 -> Peer fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747): Couldn't send request to peer fa66786862da49ae826fb519c7d89b5e. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.67:40747: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:52:43.486536 1058 consensus_peers.cc:487] T 578c441034d447f99fa557b63e643fba P 957a8a8be3f542f48e8824e581fd0f89 -> Peer fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747): Couldn't send request to peer fa66786862da49ae826fb519c7d89b5e. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.67:40747: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:52:43.498093 1190 consensus_peers.cc:487] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc -> Peer 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499): Couldn't send request to peer 75c4f59060974a3b823305b9dd339a7e. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.66:36499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250627 13:52:43.558609 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 909
W20250627 13:52:43.561625 1058 consensus_peers.cc:487] T 76a58b66489c4dd68d5c037aaffa9835 P 957a8a8be3f542f48e8824e581fd0f89 -> Peer fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747): Couldn't send request to peer fa66786862da49ae826fb519c7d89b5e. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.67:40747: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:52:43.562119 1056 consensus_peers.cc:487] T 578c441034d447f99fa557b63e643fba P 957a8a8be3f542f48e8824e581fd0f89 -> Peer 9f767210a91f44ae8b83b994131eee98 (127.0.103.65:43023): Couldn't send request to peer 9f767210a91f44ae8b83b994131eee98. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:43023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:52:43.583664 1188 connection.cc:537] server connection from 127.0.103.68:55969 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250627 13:52:43.584292 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 1040
W20250627 13:52:43.584614 1189 consensus_peers.cc:487] T 32895feb7def4132953f5a4d6bf0bbd8 P 5c28ff2fe9424379819d478fdd093cfc -> Peer 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68:34289): Couldn't send request to peer 50e8c97891d74b38b5c130d3bcbabcb6. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.68:34289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:52:43.584717 1189 consensus_peers.cc:487] T ccfdf884041949bfbfa6a09574369a13 P 5c28ff2fe9424379819d478fdd093cfc -> Peer 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68:34289): Couldn't send request to peer 50e8c97891d74b38b5c130d3bcbabcb6. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.68:34289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:52:43.588037 1190 consensus_peers.cc:487] T ccfdf884041949bfbfa6a09574369a13 P 5c28ff2fe9424379819d478fdd093cfc -> Peer 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499): Couldn't send request to peer 75c4f59060974a3b823305b9dd339a7e. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.66:36499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:52:43.589612 1189 consensus_peers.cc:487] T d146d21a389944a0b47e89e85a6fe920 P 5c28ff2fe9424379819d478fdd093cfc -> Peer 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68:34289): Couldn't send request to peer 50e8c97891d74b38b5c130d3bcbabcb6. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.68:34289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250627 13:52:43.591123 1163 generic_service.cc:196] Checking for leaks (request via RPC)
W20250627 13:52:43.605990 1190 consensus_peers.cc:487] T 32895feb7def4132953f5a4d6bf0bbd8 P 5c28ff2fe9424379819d478fdd093cfc -> Peer 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499): Couldn't send request to peer 75c4f59060974a3b823305b9dd339a7e. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.66:36499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:52:43.632900 1190 consensus_peers.cc:487] T b9b2432425534c4db188ac324295513d P 5c28ff2fe9424379819d478fdd093cfc -> Peer 75c4f59060974a3b823305b9dd339a7e (127.0.103.66:36499): Couldn't send request to peer 75c4f59060974a3b823305b9dd339a7e. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.66:36499: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:52:43.717557 1189 consensus_peers.cc:487] T b7e50b9da893466486b668fcd0aac962 P 5c28ff2fe9424379819d478fdd093cfc -> Peer 50e8c97891d74b38b5c130d3bcbabcb6 (127.0.103.68:34289): Couldn't send request to peer 50e8c97891d74b38b5c130d3bcbabcb6. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.68:34289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250627 13:52:43.725350 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 1040
I20250627 13:52:43.748881 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 1171
W20250627 13:52:43.753880 1189 consensus_peers.cc:487] T acf517032e5744da93a91d4348b6ceed P 5c28ff2fe9424379819d478fdd093cfc -> Peer fa66786862da49ae826fb519c7d89b5e (127.0.103.67:40747): Couldn't send request to peer fa66786862da49ae826fb519c7d89b5e. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.67:40747: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250627 13:52:43.755172 1294 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:52:43.885876 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 1171
I20250627 13:52:43.909288 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 425
I20250627 13:52:43.910421 486 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:52:44.057817 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 425
2025-06-27T13:52:44Z chronyd exiting
[ OK ] LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency (30342 ms)
[----------] 1 test from LocationAwareRebalancingBasicTest (30342 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1
I20250627 13:52:44.136118 413 test_util.cc:276] Using random seed: -32639344
I20250627 13:52:44.138079 413 ts_itest-base.cc:115] Starting cluster with:
I20250627 13:52:44.138160 413 ts_itest-base.cc:116] --------------
I20250627 13:52:44.138195 413 ts_itest-base.cc:117] 5 tablet servers
I20250627 13:52:44.138263 413 ts_itest-base.cc:118] 3 replicas per TS
I20250627 13:52:44.138312 413 ts_itest-base.cc:119] --------------
2025-06-27T13:52:44Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-27T13:52:44Z Disabled control of system clock
I20250627 13:52:44.165328 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.103.126:44261
--webserver_interface=127.0.103.126
--webserver_port=0
--builtin_ntp_servers=127.0.103.84:37479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.103.126:44261
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20250627 13:52:44.474972 2794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:44.475312 2794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:52:44.475426 2794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:44.483291 2794 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:52:44.483402 2794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250627 13:52:44.483477 2794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:52:44.483554 2794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250627 13:52:44.483642 2794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250627 13:52:44.494668 2794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:37479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.103.126:44261
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.103.126:44261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.103.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:52:44.496383 2794 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:52:44.498236 2794 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:52:44.508664 2800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:44.508744 2799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:44.508736 2802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:52:44.509225 2794 server_base.cc:1048] running on GCE node
I20250627 13:52:44.510179 2794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:52:44.511508 2794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:52:44.512737 2794 hybrid_clock.cc:648] HybridClock initialized: now 1751032364512678 us; error 40 us; skew 500 ppm
I20250627 13:52:44.513039 2794 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:52:44.519559 2794 webserver.cc:469] Webserver started at http://127.0.103.126:36123/ using document root <none> and password file <none>
I20250627 13:52:44.520326 2794 fs_manager.cc:362] Metadata directory not provided
I20250627 13:52:44.520452 2794 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:52:44.520776 2794 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:52:44.523180 2794 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "a8465bc57bd34c3891c7b4cd3ffe4a19"
format_stamp: "Formatted at 2025-06-27 13:52:44 on dist-test-slave-1ckg"
I20250627 13:52:44.523908 2794 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "a8465bc57bd34c3891c7b4cd3ffe4a19"
format_stamp: "Formatted at 2025-06-27 13:52:44 on dist-test-slave-1ckg"
I20250627 13:52:44.529008 2794 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20250627 13:52:44.532372 2808 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:44.533694 2794 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20250627 13:52:44.533844 2794 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a8465bc57bd34c3891c7b4cd3ffe4a19"
format_stamp: "Formatted at 2025-06-27 13:52:44 on dist-test-slave-1ckg"
I20250627 13:52:44.534039 2794 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:52:44.573347 2794 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:52:44.574244 2794 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:52:44.574499 2794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:52:44.593770 2794 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:44261
I20250627 13:52:44.593767 2859 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:44261 every 8 connection(s)
I20250627 13:52:44.595292 2794 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250627 13:52:44.599213 2860 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:44.602798 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 2794
I20250627 13:52:44.602983 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250627 13:52:44.608969 2860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8465bc57bd34c3891c7b4cd3ffe4a19: Bootstrap starting.
I20250627 13:52:44.612491 2860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8465bc57bd34c3891c7b4cd3ffe4a19: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:44.613782 2860 log.cc:826] T 00000000000000000000000000000000 P a8465bc57bd34c3891c7b4cd3ffe4a19: Log is configured to *not* fsync() on all Append() calls
I20250627 13:52:44.616277 2860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8465bc57bd34c3891c7b4cd3ffe4a19: No bootstrap required, opened a new log
I20250627 13:52:44.621042 2860 raft_consensus.cc:357] T 00000000000000000000000000000000 P a8465bc57bd34c3891c7b4cd3ffe4a19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8465bc57bd34c3891c7b4cd3ffe4a19" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 44261 } }
I20250627 13:52:44.621327 2860 raft_consensus.cc:383] T 00000000000000000000000000000000 P a8465bc57bd34c3891c7b4cd3ffe4a19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:44.621444 2860 raft_consensus.cc:738] T 00000000000000000000000000000000 P a8465bc57bd34c3891c7b4cd3ffe4a19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8465bc57bd34c3891c7b4cd3ffe4a19, State: Initialized, Role: FOLLOWER
I20250627 13:52:44.622010 2860 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8465bc57bd34c3891c7b4cd3ffe4a19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8465bc57bd34c3891c7b4cd3ffe4a19" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 44261 } }
I20250627 13:52:44.622229 2860 raft_consensus.cc:397] T 00000000000000000000000000000000 P a8465bc57bd34c3891c7b4cd3ffe4a19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:52:44.622337 2860 raft_consensus.cc:491] T 00000000000000000000000000000000 P a8465bc57bd34c3891c7b4cd3ffe4a19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:52:44.622503 2860 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a8465bc57bd34c3891c7b4cd3ffe4a19 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:44.624584 2860 raft_consensus.cc:513] T 00000000000000000000000000000000 P a8465bc57bd34c3891c7b4cd3ffe4a19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8465bc57bd34c3891c7b4cd3ffe4a19" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 44261 } }
I20250627 13:52:44.625005 2860 leader_election.cc:304] T 00000000000000000000000000000000 P a8465bc57bd34c3891c7b4cd3ffe4a19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8465bc57bd34c3891c7b4cd3ffe4a19; no voters:
I20250627 13:52:44.625463 2860 leader_election.cc:290] T 00000000000000000000000000000000 P a8465bc57bd34c3891c7b4cd3ffe4a19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250627 13:52:44.625561 2865 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a8465bc57bd34c3891c7b4cd3ffe4a19 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:44.627097 2865 raft_consensus.cc:695] T 00000000000000000000000000000000 P a8465bc57bd34c3891c7b4cd3ffe4a19 [term 1 LEADER]: Becoming Leader. State: Replica: a8465bc57bd34c3891c7b4cd3ffe4a19, State: Running, Role: LEADER
I20250627 13:52:44.627609 2860 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8465bc57bd34c3891c7b4cd3ffe4a19 [sys.catalog]: configured and running, proceeding with master startup.
I20250627 13:52:44.627557 2865 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8465bc57bd34c3891c7b4cd3ffe4a19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8465bc57bd34c3891c7b4cd3ffe4a19" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 44261 } }
I20250627 13:52:44.630518 2866 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8465bc57bd34c3891c7b4cd3ffe4a19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8465bc57bd34c3891c7b4cd3ffe4a19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8465bc57bd34c3891c7b4cd3ffe4a19" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 44261 } } }
I20250627 13:52:44.630511 2867 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8465bc57bd34c3891c7b4cd3ffe4a19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8465bc57bd34c3891c7b4cd3ffe4a19. Latest consensus state: current_term: 1 leader_uuid: "a8465bc57bd34c3891c7b4cd3ffe4a19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8465bc57bd34c3891c7b4cd3ffe4a19" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 44261 } } }
I20250627 13:52:44.630911 2866 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8465bc57bd34c3891c7b4cd3ffe4a19 [sys.catalog]: This master's current role is: LEADER
I20250627 13:52:44.630923 2867 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8465bc57bd34c3891c7b4cd3ffe4a19 [sys.catalog]: This master's current role is: LEADER
I20250627 13:52:44.631850 2874 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250627 13:52:44.637593 2874 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250627 13:52:44.645808 2874 catalog_manager.cc:1349] Generated new cluster ID: e0758c04458f4845b00afb8e949012b1
I20250627 13:52:44.645931 2874 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250627 13:52:44.664193 2874 catalog_manager.cc:1372] Generated new certificate authority record
I20250627 13:52:44.665850 2874 catalog_manager.cc:1506] Loading token signing keys...
I20250627 13:52:44.688735 2874 catalog_manager.cc:5955] T 00000000000000000000000000000000 P a8465bc57bd34c3891c7b4cd3ffe4a19: Generated new TSK 0
I20250627 13:52:44.689729 2874 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250627 13:52:44.703421 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.65:0
--local_ip_for_outbound_sockets=127.0.103.65
--webserver_interface=127.0.103.65
--webserver_port=0
--tserver_master_addrs=127.0.103.126:44261
--builtin_ntp_servers=127.0.103.84:37479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250627 13:52:45.049254 2884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:45.049532 2884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:52:45.049614 2884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:45.057080 2884 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:52:45.057191 2884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:52:45.057356 2884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.65
I20250627 13:52:45.066354 2884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:37479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.103.65
--webserver_port=0
--tserver_master_addrs=127.0.103.126:44261
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.65
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:52:45.067989 2884 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:52:45.069779 2884 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:52:45.080958 2889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:45.080958 2890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:45.081029 2892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:52:45.081810 2884 server_base.cc:1048] running on GCE node
I20250627 13:52:45.082331 2884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:52:45.083141 2884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:52:45.084347 2884 hybrid_clock.cc:648] HybridClock initialized: now 1751032365084276 us; error 45 us; skew 500 ppm
I20250627 13:52:45.084609 2884 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:52:45.086706 2884 webserver.cc:469] Webserver started at http://127.0.103.65:36333/ using document root <none> and password file <none>
I20250627 13:52:45.087424 2884 fs_manager.cc:362] Metadata directory not provided
I20250627 13:52:45.087529 2884 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:52:45.087855 2884 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:52:45.090152 2884 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f120ac473ba94edf8d02caa27dc070ee"
format_stamp: "Formatted at 2025-06-27 13:52:45 on dist-test-slave-1ckg"
I20250627 13:52:45.090911 2884 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f120ac473ba94edf8d02caa27dc070ee"
format_stamp: "Formatted at 2025-06-27 13:52:45 on dist-test-slave-1ckg"
I20250627 13:52:45.095921 2884 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250627 13:52:45.098951 2898 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:45.100112 2884 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250627 13:52:45.100265 2884 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f120ac473ba94edf8d02caa27dc070ee"
format_stamp: "Formatted at 2025-06-27 13:52:45 on dist-test-slave-1ckg"
I20250627 13:52:45.100428 2884 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:52:45.114792 2884 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:52:45.115531 2884 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:52:45.115777 2884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:52:45.116575 2884 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:52:45.118074 2884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:52:45.118151 2884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:45.118285 2884 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:52:45.118340 2884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:45.149816 2884 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:44165
I20250627 13:52:45.149859 3010 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:44165 every 8 connection(s)
I20250627 13:52:45.151182 2884 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250627 13:52:45.153766 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 2884
I20250627 13:52:45.153980 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250627 13:52:45.157402 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.66:0
--local_ip_for_outbound_sockets=127.0.103.66
--webserver_interface=127.0.103.66
--webserver_port=0
--tserver_master_addrs=127.0.103.126:44261
--builtin_ntp_servers=127.0.103.84:37479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:52:45.164750 3011 heartbeater.cc:344] Connected to a master server at 127.0.103.126:44261
I20250627 13:52:45.165102 3011 heartbeater.cc:461] Registering TS with master...
I20250627 13:52:45.165995 3011 heartbeater.cc:507] Master 127.0.103.126:44261 requested a full tablet report, sending...
I20250627 13:52:45.168067 2825 ts_manager.cc:194] Registered new tserver with Master: f120ac473ba94edf8d02caa27dc070ee (127.0.103.65:44165)
I20250627 13:52:45.169531 2825 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.65:54175
W20250627 13:52:45.463615 3015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:45.463961 3015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:52:45.464097 3015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:45.471490 3015 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:52:45.471625 3015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:52:45.471825 3015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.66
I20250627 13:52:45.480558 3015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:37479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.103.66
--webserver_port=0
--tserver_master_addrs=127.0.103.126:44261
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.66
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:52:45.482177 3015 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:52:45.483958 3015 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:52:45.494710 3021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:45.495296 3023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:45.495371 3020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:52:45.495685 3015 server_base.cc:1048] running on GCE node
I20250627 13:52:45.496435 3015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:52:45.497336 3015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:52:45.498574 3015 hybrid_clock.cc:648] HybridClock initialized: now 1751032365498512 us; error 31 us; skew 500 ppm
I20250627 13:52:45.498823 3015 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:52:45.501005 3015 webserver.cc:469] Webserver started at http://127.0.103.66:43871/ using document root <none> and password file <none>
I20250627 13:52:45.501688 3015 fs_manager.cc:362] Metadata directory not provided
I20250627 13:52:45.501825 3015 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:52:45.502128 3015 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:52:45.506825 3015 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "aad7fce099ea4b10a44c494a2777e1fb"
format_stamp: "Formatted at 2025-06-27 13:52:45 on dist-test-slave-1ckg"
I20250627 13:52:45.507508 3015 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "aad7fce099ea4b10a44c494a2777e1fb"
format_stamp: "Formatted at 2025-06-27 13:52:45 on dist-test-slave-1ckg"
I20250627 13:52:45.512408 3015 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20250627 13:52:45.515553 3029 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:45.516772 3015 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250627 13:52:45.516911 3015 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "aad7fce099ea4b10a44c494a2777e1fb"
format_stamp: "Formatted at 2025-06-27 13:52:45 on dist-test-slave-1ckg"
I20250627 13:52:45.517094 3015 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:52:45.547703 3015 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:52:45.548573 3015 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:52:45.548838 3015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:52:45.549777 3015 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:52:45.551316 3015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:52:45.551388 3015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:45.551505 3015 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:52:45.551551 3015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:45.582844 3015 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:46047
I20250627 13:52:45.582922 3141 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:46047 every 8 connection(s)
I20250627 13:52:45.584112 3015 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250627 13:52:45.586238 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 3015
I20250627 13:52:45.586429 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250627 13:52:45.589202 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.67:0
--local_ip_for_outbound_sockets=127.0.103.67
--webserver_interface=127.0.103.67
--webserver_port=0
--tserver_master_addrs=127.0.103.126:44261
--builtin_ntp_servers=127.0.103.84:37479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:52:45.597088 3142 heartbeater.cc:344] Connected to a master server at 127.0.103.126:44261
I20250627 13:52:45.597430 3142 heartbeater.cc:461] Registering TS with master...
I20250627 13:52:45.598338 3142 heartbeater.cc:507] Master 127.0.103.126:44261 requested a full tablet report, sending...
I20250627 13:52:45.599870 2825 ts_manager.cc:194] Registered new tserver with Master: aad7fce099ea4b10a44c494a2777e1fb (127.0.103.66:46047)
I20250627 13:52:45.601137 2825 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.66:55113
W20250627 13:52:45.893206 3145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:45.893533 3145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:52:45.893710 3145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:45.901479 3145 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:52:45.901587 3145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:52:45.901799 3145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.67
I20250627 13:52:45.911271 3145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:37479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.103.67
--webserver_port=0
--tserver_master_addrs=127.0.103.126:44261
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.67
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:52:45.912925 3145 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:52:45.914764 3145 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:52:45.926147 3153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:45.926231 3150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:45.926231 3151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:52:45.927289 3145 server_base.cc:1048] running on GCE node
I20250627 13:52:45.927913 3145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:52:45.928871 3145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:52:45.930081 3145 hybrid_clock.cc:648] HybridClock initialized: now 1751032365930033 us; error 45 us; skew 500 ppm
I20250627 13:52:45.930367 3145 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:52:45.932822 3145 webserver.cc:469] Webserver started at http://127.0.103.67:36449/ using document root <none> and password file <none>
I20250627 13:52:45.933553 3145 fs_manager.cc:362] Metadata directory not provided
I20250627 13:52:45.933713 3145 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:52:45.934065 3145 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:52:45.936468 3145 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "158de984542f427b8917ccda989ce408"
format_stamp: "Formatted at 2025-06-27 13:52:45 on dist-test-slave-1ckg"
I20250627 13:52:45.937314 3145 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "158de984542f427b8917ccda989ce408"
format_stamp: "Formatted at 2025-06-27 13:52:45 on dist-test-slave-1ckg"
I20250627 13:52:45.942854 3145 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20250627 13:52:45.946223 3159 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:45.947576 3145 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250627 13:52:45.947710 3145 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "158de984542f427b8917ccda989ce408"
format_stamp: "Formatted at 2025-06-27 13:52:45 on dist-test-slave-1ckg"
I20250627 13:52:45.947860 3145 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:52:45.961855 3145 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:52:45.962644 3145 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:52:45.962879 3145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:52:45.963788 3145 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:52:45.965341 3145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:52:45.965413 3145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:45.965508 3145 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:52:45.965551 3145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:45.997732 3145 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:45015
I20250627 13:52:45.997781 3271 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:45015 every 8 connection(s)
I20250627 13:52:45.999073 3145 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250627 13:52:46.007059 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 3145
I20250627 13:52:46.007237 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250627 13:52:46.010571 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.68:0
--local_ip_for_outbound_sockets=127.0.103.68
--webserver_interface=127.0.103.68
--webserver_port=0
--tserver_master_addrs=127.0.103.126:44261
--builtin_ntp_servers=127.0.103.84:37479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:52:46.012013 3272 heartbeater.cc:344] Connected to a master server at 127.0.103.126:44261
I20250627 13:52:46.012442 3272 heartbeater.cc:461] Registering TS with master...
I20250627 13:52:46.013548 3272 heartbeater.cc:507] Master 127.0.103.126:44261 requested a full tablet report, sending...
I20250627 13:52:46.015224 2825 ts_manager.cc:194] Registered new tserver with Master: 158de984542f427b8917ccda989ce408 (127.0.103.67:45015)
I20250627 13:52:46.015990 2825 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.67:45677
I20250627 13:52:46.172312 3011 heartbeater.cc:499] Master 127.0.103.126:44261 was elected leader, sending a full tablet report...
W20250627 13:52:46.324487 3275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:46.324790 3275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:52:46.324920 3275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:46.332927 3275 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:52:46.333048 3275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:52:46.333240 3275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.68
I20250627 13:52:46.342648 3275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:37479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.103.68
--webserver_port=0
--tserver_master_addrs=127.0.103.126:44261
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.68
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:52:46.344281 3275 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:52:46.346094 3275 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:52:46.357470 3280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:46.357403 3283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:46.357395 3281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:52:46.358444 3275 server_base.cc:1048] running on GCE node
I20250627 13:52:46.359158 3275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:52:46.360056 3275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:52:46.361279 3275 hybrid_clock.cc:648] HybridClock initialized: now 1751032366361208 us; error 45 us; skew 500 ppm
I20250627 13:52:46.361580 3275 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:52:46.363848 3275 webserver.cc:469] Webserver started at http://127.0.103.68:44119/ using document root <none> and password file <none>
I20250627 13:52:46.364571 3275 fs_manager.cc:362] Metadata directory not provided
I20250627 13:52:46.364691 3275 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:52:46.365020 3275 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:52:46.367312 3275 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "32e5990a6e4c4a1482546c280dc3df5d"
format_stamp: "Formatted at 2025-06-27 13:52:46 on dist-test-slave-1ckg"
I20250627 13:52:46.368075 3275 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "32e5990a6e4c4a1482546c280dc3df5d"
format_stamp: "Formatted at 2025-06-27 13:52:46 on dist-test-slave-1ckg"
I20250627 13:52:46.373008 3275 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20250627 13:52:46.376128 3289 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:46.377385 3275 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250627 13:52:46.377530 3275 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "32e5990a6e4c4a1482546c280dc3df5d"
format_stamp: "Formatted at 2025-06-27 13:52:46 on dist-test-slave-1ckg"
I20250627 13:52:46.377724 3275 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:52:46.394506 3275 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:52:46.395202 3275 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:52:46.395412 3275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:52:46.396265 3275 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:52:46.397768 3275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:52:46.397848 3275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:46.397958 3275 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:52:46.398028 3275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:46.429375 3275 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:38711
I20250627 13:52:46.429433 3401 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:38711 every 8 connection(s)
I20250627 13:52:46.430706 3275 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250627 13:52:46.438987 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 3275
I20250627 13:52:46.439210 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250627 13:52:46.442216 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.69:0
--local_ip_for_outbound_sockets=127.0.103.69
--webserver_interface=127.0.103.69
--webserver_port=0
--tserver_master_addrs=127.0.103.126:44261
--builtin_ntp_servers=127.0.103.84:37479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:52:46.443496 3402 heartbeater.cc:344] Connected to a master server at 127.0.103.126:44261
I20250627 13:52:46.443841 3402 heartbeater.cc:461] Registering TS with master...
I20250627 13:52:46.444708 3402 heartbeater.cc:507] Master 127.0.103.126:44261 requested a full tablet report, sending...
I20250627 13:52:46.446256 2825 ts_manager.cc:194] Registered new tserver with Master: 32e5990a6e4c4a1482546c280dc3df5d (127.0.103.68:38711)
I20250627 13:52:46.447011 2825 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.68:53295
I20250627 13:52:46.603993 3142 heartbeater.cc:499] Master 127.0.103.126:44261 was elected leader, sending a full tablet report...
W20250627 13:52:46.764047 3405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:46.764415 3405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:52:46.764539 3405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:46.772048 3405 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:52:46.772182 3405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:52:46.772378 3405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.69
I20250627 13:52:46.782110 3405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:37479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.103.69
--webserver_port=0
--tserver_master_addrs=127.0.103.126:44261
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.69
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:52:46.783831 3405 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:52:46.785631 3405 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:52:46.797216 3411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:46.797384 3410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:46.797292 3413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:52:46.798107 3405 server_base.cc:1048] running on GCE node
I20250627 13:52:46.798676 3405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:52:46.799553 3405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:52:46.800756 3405 hybrid_clock.cc:648] HybridClock initialized: now 1751032366800695 us; error 38 us; skew 500 ppm
I20250627 13:52:46.801054 3405 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:52:46.803356 3405 webserver.cc:469] Webserver started at http://127.0.103.69:37627/ using document root <none> and password file <none>
I20250627 13:52:46.804060 3405 fs_manager.cc:362] Metadata directory not provided
I20250627 13:52:46.804177 3405 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:52:46.804507 3405 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:52:46.806932 3405 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "8dafda988e794266a4f78efdcadee288"
format_stamp: "Formatted at 2025-06-27 13:52:46 on dist-test-slave-1ckg"
I20250627 13:52:46.807699 3405 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "8dafda988e794266a4f78efdcadee288"
format_stamp: "Formatted at 2025-06-27 13:52:46 on dist-test-slave-1ckg"
I20250627 13:52:46.812747 3405 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250627 13:52:46.815927 3419 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:46.817201 3405 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20250627 13:52:46.817373 3405 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "8dafda988e794266a4f78efdcadee288"
format_stamp: "Formatted at 2025-06-27 13:52:46 on dist-test-slave-1ckg"
I20250627 13:52:46.817561 3405 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:52:46.846273 3405 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:52:46.847101 3405 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:52:46.847373 3405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:52:46.848284 3405 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:52:46.849996 3405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:52:46.850081 3405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:46.850210 3405 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:52:46.850270 3405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:46.882203 3405 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.69:33325
I20250627 13:52:46.882282 3531 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.69:33325 every 8 connection(s)
I20250627 13:52:46.883697 3405 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250627 13:52:46.889967 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 3405
I20250627 13:52:46.890164 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250627 13:52:46.896754 3532 heartbeater.cc:344] Connected to a master server at 127.0.103.126:44261
I20250627 13:52:46.897094 3532 heartbeater.cc:461] Registering TS with master...
I20250627 13:52:46.897997 3532 heartbeater.cc:507] Master 127.0.103.126:44261 requested a full tablet report, sending...
I20250627 13:52:46.899271 2825 ts_manager.cc:194] Registered new tserver with Master: 8dafda988e794266a4f78efdcadee288 (127.0.103.69:33325)
I20250627 13:52:46.899972 2825 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.69:46163
I20250627 13:52:46.905539 413 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250627 13:52:46.921470 2825 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44260:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250627 13:52:46.954492 3207 tablet_service.cc:1468] Processing CreateTablet for tablet 6680a8603fb2489a9ede711c4eec45c1 (DEFAULT_TABLE table=TestTable [id=351b02e529db484c85a0e5588955fc5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:46.956050 3207 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6680a8603fb2489a9ede711c4eec45c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:46.959233 3467 tablet_service.cc:1468] Processing CreateTablet for tablet 6680a8603fb2489a9ede711c4eec45c1 (DEFAULT_TABLE table=TestTable [id=351b02e529db484c85a0e5588955fc5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:46.960664 3467 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6680a8603fb2489a9ede711c4eec45c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:46.965782 2946 tablet_service.cc:1468] Processing CreateTablet for tablet 6680a8603fb2489a9ede711c4eec45c1 (DEFAULT_TABLE table=TestTable [id=351b02e529db484c85a0e5588955fc5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:46.967222 2946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6680a8603fb2489a9ede711c4eec45c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:46.968400 3550 tablet_bootstrap.cc:492] T 6680a8603fb2489a9ede711c4eec45c1 P 158de984542f427b8917ccda989ce408: Bootstrap starting.
I20250627 13:52:46.972029 3551 tablet_bootstrap.cc:492] T 6680a8603fb2489a9ede711c4eec45c1 P 8dafda988e794266a4f78efdcadee288: Bootstrap starting.
I20250627 13:52:46.972872 3550 tablet_bootstrap.cc:654] T 6680a8603fb2489a9ede711c4eec45c1 P 158de984542f427b8917ccda989ce408: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:46.974025 3550 log.cc:826] T 6680a8603fb2489a9ede711c4eec45c1 P 158de984542f427b8917ccda989ce408: Log is configured to *not* fsync() on all Append() calls
I20250627 13:52:46.975664 3551 tablet_bootstrap.cc:654] T 6680a8603fb2489a9ede711c4eec45c1 P 8dafda988e794266a4f78efdcadee288: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:46.976423 3550 tablet_bootstrap.cc:492] T 6680a8603fb2489a9ede711c4eec45c1 P 158de984542f427b8917ccda989ce408: No bootstrap required, opened a new log
I20250627 13:52:46.976670 3550 ts_tablet_manager.cc:1397] T 6680a8603fb2489a9ede711c4eec45c1 P 158de984542f427b8917ccda989ce408: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20250627 13:52:46.976930 3551 log.cc:826] T 6680a8603fb2489a9ede711c4eec45c1 P 8dafda988e794266a4f78efdcadee288: Log is configured to *not* fsync() on all Append() calls
I20250627 13:52:46.979975 3554 tablet_bootstrap.cc:492] T 6680a8603fb2489a9ede711c4eec45c1 P f120ac473ba94edf8d02caa27dc070ee: Bootstrap starting.
I20250627 13:52:46.980911 3551 tablet_bootstrap.cc:492] T 6680a8603fb2489a9ede711c4eec45c1 P 8dafda988e794266a4f78efdcadee288: No bootstrap required, opened a new log
I20250627 13:52:46.981177 3551 ts_tablet_manager.cc:1397] T 6680a8603fb2489a9ede711c4eec45c1 P 8dafda988e794266a4f78efdcadee288: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20250627 13:52:46.983814 3554 tablet_bootstrap.cc:654] T 6680a8603fb2489a9ede711c4eec45c1 P f120ac473ba94edf8d02caa27dc070ee: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:46.983873 3550 raft_consensus.cc:357] T 6680a8603fb2489a9ede711c4eec45c1 P 158de984542f427b8917ccda989ce408 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158de984542f427b8917ccda989ce408" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45015 } } peers { permanent_uuid: "8dafda988e794266a4f78efdcadee288" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33325 } } peers { permanent_uuid: "f120ac473ba94edf8d02caa27dc070ee" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44165 } }
I20250627 13:52:46.984280 3550 raft_consensus.cc:383] T 6680a8603fb2489a9ede711c4eec45c1 P 158de984542f427b8917ccda989ce408 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:46.984422 3550 raft_consensus.cc:738] T 6680a8603fb2489a9ede711c4eec45c1 P 158de984542f427b8917ccda989ce408 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158de984542f427b8917ccda989ce408, State: Initialized, Role: FOLLOWER
I20250627 13:52:46.985162 3554 log.cc:826] T 6680a8603fb2489a9ede711c4eec45c1 P f120ac473ba94edf8d02caa27dc070ee: Log is configured to *not* fsync() on all Append() calls
I20250627 13:52:46.985112 3550 consensus_queue.cc:260] T 6680a8603fb2489a9ede711c4eec45c1 P 158de984542f427b8917ccda989ce408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158de984542f427b8917ccda989ce408" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45015 } } peers { permanent_uuid: "8dafda988e794266a4f78efdcadee288" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33325 } } peers { permanent_uuid: "f120ac473ba94edf8d02caa27dc070ee" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44165 } }
I20250627 13:52:46.986331 3272 heartbeater.cc:499] Master 127.0.103.126:44261 was elected leader, sending a full tablet report...
I20250627 13:52:46.987298 3550 ts_tablet_manager.cc:1428] T 6680a8603fb2489a9ede711c4eec45c1 P 158de984542f427b8917ccda989ce408: Time spent starting tablet: real 0.010s user 0.005s sys 0.006s
I20250627 13:52:46.988118 3551 raft_consensus.cc:357] T 6680a8603fb2489a9ede711c4eec45c1 P 8dafda988e794266a4f78efdcadee288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158de984542f427b8917ccda989ce408" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45015 } } peers { permanent_uuid: "8dafda988e794266a4f78efdcadee288" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33325 } } peers { permanent_uuid: "f120ac473ba94edf8d02caa27dc070ee" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44165 } }
I20250627 13:52:46.988492 3551 raft_consensus.cc:383] T 6680a8603fb2489a9ede711c4eec45c1 P 8dafda988e794266a4f78efdcadee288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:46.988616 3551 raft_consensus.cc:738] T 6680a8603fb2489a9ede711c4eec45c1 P 8dafda988e794266a4f78efdcadee288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dafda988e794266a4f78efdcadee288, State: Initialized, Role: FOLLOWER
I20250627 13:52:46.989295 3551 consensus_queue.cc:260] T 6680a8603fb2489a9ede711c4eec45c1 P 8dafda988e794266a4f78efdcadee288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158de984542f427b8917ccda989ce408" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45015 } } peers { permanent_uuid: "8dafda988e794266a4f78efdcadee288" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33325 } } peers { permanent_uuid: "f120ac473ba94edf8d02caa27dc070ee" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44165 } }
I20250627 13:52:46.990015 3554 tablet_bootstrap.cc:492] T 6680a8603fb2489a9ede711c4eec45c1 P f120ac473ba94edf8d02caa27dc070ee: No bootstrap required, opened a new log
I20250627 13:52:46.990367 3554 ts_tablet_manager.cc:1397] T 6680a8603fb2489a9ede711c4eec45c1 P f120ac473ba94edf8d02caa27dc070ee: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20250627 13:52:46.995903 3554 raft_consensus.cc:357] T 6680a8603fb2489a9ede711c4eec45c1 P f120ac473ba94edf8d02caa27dc070ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158de984542f427b8917ccda989ce408" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45015 } } peers { permanent_uuid: "8dafda988e794266a4f78efdcadee288" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33325 } } peers { permanent_uuid: "f120ac473ba94edf8d02caa27dc070ee" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44165 } }
I20250627 13:52:46.996196 3554 raft_consensus.cc:383] T 6680a8603fb2489a9ede711c4eec45c1 P f120ac473ba94edf8d02caa27dc070ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:46.996338 3554 raft_consensus.cc:738] T 6680a8603fb2489a9ede711c4eec45c1 P f120ac473ba94edf8d02caa27dc070ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f120ac473ba94edf8d02caa27dc070ee, State: Initialized, Role: FOLLOWER
I20250627 13:52:46.996448 3532 heartbeater.cc:499] Master 127.0.103.126:44261 was elected leader, sending a full tablet report...
I20250627 13:52:46.996937 3554 consensus_queue.cc:260] T 6680a8603fb2489a9ede711c4eec45c1 P f120ac473ba94edf8d02caa27dc070ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158de984542f427b8917ccda989ce408" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45015 } } peers { permanent_uuid: "8dafda988e794266a4f78efdcadee288" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33325 } } peers { permanent_uuid: "f120ac473ba94edf8d02caa27dc070ee" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44165 } }
I20250627 13:52:46.997462 3551 ts_tablet_manager.cc:1428] T 6680a8603fb2489a9ede711c4eec45c1 P 8dafda988e794266a4f78efdcadee288: Time spent starting tablet: real 0.016s user 0.008s sys 0.006s
I20250627 13:52:46.999009 3554 ts_tablet_manager.cc:1428] T 6680a8603fb2489a9ede711c4eec45c1 P f120ac473ba94edf8d02caa27dc070ee: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
W20250627 13:52:47.001179 3273 tablet.cc:2378] T 6680a8603fb2489a9ede711c4eec45c1 P 158de984542f427b8917ccda989ce408: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:52:47.108503 3555 raft_consensus.cc:491] T 6680a8603fb2489a9ede711c4eec45c1 P 158de984542f427b8917ccda989ce408 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:47.108759 3555 raft_consensus.cc:513] T 6680a8603fb2489a9ede711c4eec45c1 P 158de984542f427b8917ccda989ce408 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158de984542f427b8917ccda989ce408" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45015 } } peers { permanent_uuid: "8dafda988e794266a4f78efdcadee288" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33325 } } peers { permanent_uuid: "f120ac473ba94edf8d02caa27dc070ee" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44165 } }
I20250627 13:52:47.110647 3555 leader_election.cc:290] T 6680a8603fb2489a9ede711c4eec45c1 P 158de984542f427b8917ccda989ce408 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dafda988e794266a4f78efdcadee288 (127.0.103.69:33325), f120ac473ba94edf8d02caa27dc070ee (127.0.103.65:44165)
I20250627 13:52:47.117760 3487 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6680a8603fb2489a9ede711c4eec45c1" candidate_uuid: "158de984542f427b8917ccda989ce408" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dafda988e794266a4f78efdcadee288" is_pre_election: true
I20250627 13:52:47.117969 2966 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6680a8603fb2489a9ede711c4eec45c1" candidate_uuid: "158de984542f427b8917ccda989ce408" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f120ac473ba94edf8d02caa27dc070ee" is_pre_election: true
I20250627 13:52:47.118273 3487 raft_consensus.cc:2466] T 6680a8603fb2489a9ede711c4eec45c1 P 8dafda988e794266a4f78efdcadee288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 158de984542f427b8917ccda989ce408 in term 0.
I20250627 13:52:47.118427 2966 raft_consensus.cc:2466] T 6680a8603fb2489a9ede711c4eec45c1 P f120ac473ba94edf8d02caa27dc070ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 158de984542f427b8917ccda989ce408 in term 0.
I20250627 13:52:47.118875 3161 leader_election.cc:304] T 6680a8603fb2489a9ede711c4eec45c1 P 158de984542f427b8917ccda989ce408 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158de984542f427b8917ccda989ce408, 8dafda988e794266a4f78efdcadee288; no voters:
I20250627 13:52:47.119216 3555 raft_consensus.cc:2802] T 6680a8603fb2489a9ede711c4eec45c1 P 158de984542f427b8917ccda989ce408 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:47.119369 3555 raft_consensus.cc:491] T 6680a8603fb2489a9ede711c4eec45c1 P 158de984542f427b8917ccda989ce408 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:47.119484 3555 raft_consensus.cc:3058] T 6680a8603fb2489a9ede711c4eec45c1 P 158de984542f427b8917ccda989ce408 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:47.121574 3555 raft_consensus.cc:513] T 6680a8603fb2489a9ede711c4eec45c1 P 158de984542f427b8917ccda989ce408 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158de984542f427b8917ccda989ce408" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45015 } } peers { permanent_uuid: "8dafda988e794266a4f78efdcadee288" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33325 } } peers { permanent_uuid: "f120ac473ba94edf8d02caa27dc070ee" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44165 } }
I20250627 13:52:47.122273 3555 leader_election.cc:290] T 6680a8603fb2489a9ede711c4eec45c1 P 158de984542f427b8917ccda989ce408 [CANDIDATE]: Term 1 election: Requested vote from peers 8dafda988e794266a4f78efdcadee288 (127.0.103.69:33325), f120ac473ba94edf8d02caa27dc070ee (127.0.103.65:44165)
I20250627 13:52:47.122805 3487 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6680a8603fb2489a9ede711c4eec45c1" candidate_uuid: "158de984542f427b8917ccda989ce408" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dafda988e794266a4f78efdcadee288"
I20250627 13:52:47.122810 2966 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6680a8603fb2489a9ede711c4eec45c1" candidate_uuid: "158de984542f427b8917ccda989ce408" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f120ac473ba94edf8d02caa27dc070ee"
I20250627 13:52:47.122983 3487 raft_consensus.cc:3058] T 6680a8603fb2489a9ede711c4eec45c1 P 8dafda988e794266a4f78efdcadee288 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:47.123023 2966 raft_consensus.cc:3058] T 6680a8603fb2489a9ede711c4eec45c1 P f120ac473ba94edf8d02caa27dc070ee [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:47.125133 3487 raft_consensus.cc:2466] T 6680a8603fb2489a9ede711c4eec45c1 P 8dafda988e794266a4f78efdcadee288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158de984542f427b8917ccda989ce408 in term 1.
I20250627 13:52:47.125301 2966 raft_consensus.cc:2466] T 6680a8603fb2489a9ede711c4eec45c1 P f120ac473ba94edf8d02caa27dc070ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158de984542f427b8917ccda989ce408 in term 1.
I20250627 13:52:47.125586 3161 leader_election.cc:304] T 6680a8603fb2489a9ede711c4eec45c1 P 158de984542f427b8917ccda989ce408 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158de984542f427b8917ccda989ce408, 8dafda988e794266a4f78efdcadee288; no voters:
I20250627 13:52:47.125900 3555 raft_consensus.cc:2802] T 6680a8603fb2489a9ede711c4eec45c1 P 158de984542f427b8917ccda989ce408 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:47.126338 3555 raft_consensus.cc:695] T 6680a8603fb2489a9ede711c4eec45c1 P 158de984542f427b8917ccda989ce408 [term 1 LEADER]: Becoming Leader. State: Replica: 158de984542f427b8917ccda989ce408, State: Running, Role: LEADER
I20250627 13:52:47.126788 3555 consensus_queue.cc:237] T 6680a8603fb2489a9ede711c4eec45c1 P 158de984542f427b8917ccda989ce408 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158de984542f427b8917ccda989ce408" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45015 } } peers { permanent_uuid: "8dafda988e794266a4f78efdcadee288" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33325 } } peers { permanent_uuid: "f120ac473ba94edf8d02caa27dc070ee" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44165 } }
I20250627 13:52:47.131763 2824 catalog_manager.cc:5582] T 6680a8603fb2489a9ede711c4eec45c1 P 158de984542f427b8917ccda989ce408 reported cstate change: term changed from 0 to 1, leader changed from <none> to 158de984542f427b8917ccda989ce408 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "158de984542f427b8917ccda989ce408" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158de984542f427b8917ccda989ce408" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dafda988e794266a4f78efdcadee288" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f120ac473ba94edf8d02caa27dc070ee" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44165 } health_report { overall_health: UNKNOWN } } }
W20250627 13:52:47.135308 3533 tablet.cc:2378] T 6680a8603fb2489a9ede711c4eec45c1 P 8dafda988e794266a4f78efdcadee288: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250627 13:52:47.153991 3012 tablet.cc:2378] T 6680a8603fb2489a9ede711c4eec45c1 P f120ac473ba94edf8d02caa27dc070ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:52:47.188624 413 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250627 13:52:47.191869 413 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f120ac473ba94edf8d02caa27dc070ee to finish bootstrapping
I20250627 13:52:47.202987 413 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 158de984542f427b8917ccda989ce408 to finish bootstrapping
I20250627 13:52:47.212536 413 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 8dafda988e794266a4f78efdcadee288 to finish bootstrapping
I20250627 13:52:47.449633 3402 heartbeater.cc:499] Master 127.0.103.126:44261 was elected leader, sending a full tablet report...
I20250627 13:52:47.563692 3555 consensus_queue.cc:1035] T 6680a8603fb2489a9ede711c4eec45c1 P 158de984542f427b8917ccda989ce408 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f120ac473ba94edf8d02caa27dc070ee" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:47.574115 3561 consensus_queue.cc:1035] T 6680a8603fb2489a9ede711c4eec45c1 P 158de984542f427b8917ccda989ce408 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dafda988e794266a4f78efdcadee288" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:47.679778 3207 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:52:47.698992 2946 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:52:47.699865 3077 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:52:47.700794 3337 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:52:47.700796 3467 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:52:47.864789 3207 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:52:47.878253 3337 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:52:47.884806 3467 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:52:47.886382 3077 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:52:47.903587 2946 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:52:48.496126 3207 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:52:48.496496 3337 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:52:48.499061 3467 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:52:48.506052 3077 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:52:48.523633 2946 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:52:48.674526 3207 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:52:48.681221 3337 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:52:48.682746 3077 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:52:48.683070 3467 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:52:48.708099 2946 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:52:48.858224 3337 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:52:48.859943 3207 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:52:48.866294 3467 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:52:48.879609 2946 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:52:48.880384 3077 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:52:49.033033 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 2884
I20250627 13:52:49.034482 3006 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:52:49.127569 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 2884
I20250627 13:52:49.145527 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 3015
I20250627 13:52:49.151693 3136 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:52:49.239096 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 3015
I20250627 13:52:49.256134 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 3145
I20250627 13:52:49.257202 3266 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:52:49.350989 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 3145
I20250627 13:52:49.369463 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 3275
I20250627 13:52:49.375906 3397 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:52:49.464732 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 3275
I20250627 13:52:49.481457 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 3405
I20250627 13:52:49.482517 3526 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:52:49.577289 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 3405
I20250627 13:52:49.594730 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 2794
I20250627 13:52:49.595892 2855 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:52:49.705727 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 2794
2025-06-27T13:52:49Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1 (5613 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5613 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/3
I20250627 13:52:49.749289 413 test_util.cc:276] Using random seed: -27026163
I20250627 13:52:49.751255 413 ts_itest-base.cc:115] Starting cluster with:
I20250627 13:52:49.751329 413 ts_itest-base.cc:116] --------------
I20250627 13:52:49.751398 413 ts_itest-base.cc:117] 7 tablet servers
I20250627 13:52:49.751453 413 ts_itest-base.cc:118] 2 replicas per TS
I20250627 13:52:49.751518 413 ts_itest-base.cc:119] --------------
2025-06-27T13:52:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-27T13:52:49Z Disabled control of system clock
I20250627 13:52:49.774125 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.103.126:39869
--webserver_interface=127.0.103.126
--webserver_port=0
--builtin_ntp_servers=127.0.103.84:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.103.126:39869
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20250627 13:52:50.075456 3761 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20250627 13:52:50.075840 3761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:50.075927 3761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:52:50.076045 3761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:50.083693 3761 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:52:50.083866 3761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250627 13:52:50.083959 3761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:52:50.084043 3761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250627 13:52:50.084112 3761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250627 13:52:50.093117 3761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.103.126:39869
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.103.126:39869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.103.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:52:50.094774 3761 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:52:50.096560 3761 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:52:50.106928 3767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:50.107079 3769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:50.106884 3766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:52:50.108131 3761 server_base.cc:1048] running on GCE node
I20250627 13:52:50.108880 3761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:52:50.110224 3761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:52:50.111419 3761 hybrid_clock.cc:648] HybridClock initialized: now 1751032370111369 us; error 41 us; skew 500 ppm
I20250627 13:52:50.111760 3761 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:52:50.117795 3761 webserver.cc:469] Webserver started at http://127.0.103.126:35899/ using document root <none> and password file <none>
I20250627 13:52:50.118577 3761 fs_manager.cc:362] Metadata directory not provided
I20250627 13:52:50.118685 3761 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:52:50.119048 3761 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:52:50.121371 3761 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "bcc04dadc3364622b3a0e2d593b2dc60"
format_stamp: "Formatted at 2025-06-27 13:52:50 on dist-test-slave-1ckg"
I20250627 13:52:50.122161 3761 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "bcc04dadc3364622b3a0e2d593b2dc60"
format_stamp: "Formatted at 2025-06-27 13:52:50 on dist-test-slave-1ckg"
I20250627 13:52:50.126943 3761 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250627 13:52:50.129890 3775 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:50.131044 3761 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250627 13:52:50.131186 3761 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/wal
uuid: "bcc04dadc3364622b3a0e2d593b2dc60"
format_stamp: "Formatted at 2025-06-27 13:52:50 on dist-test-slave-1ckg"
I20250627 13:52:50.131351 3761 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:52:50.165443 3761 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:52:50.166308 3761 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:52:50.166556 3761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:52:50.185120 3761 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:39869
I20250627 13:52:50.185142 3826 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:39869 every 8 connection(s)
I20250627 13:52:50.186620 3761 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250627 13:52:50.189852 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 3761
I20250627 13:52:50.190040 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250627 13:52:50.190413 3827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:50.200531 3827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcc04dadc3364622b3a0e2d593b2dc60: Bootstrap starting.
I20250627 13:52:50.204052 3827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcc04dadc3364622b3a0e2d593b2dc60: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:50.205061 3827 log.cc:826] T 00000000000000000000000000000000 P bcc04dadc3364622b3a0e2d593b2dc60: Log is configured to *not* fsync() on all Append() calls
I20250627 13:52:50.207451 3827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcc04dadc3364622b3a0e2d593b2dc60: No bootstrap required, opened a new log
I20250627 13:52:50.212013 3827 raft_consensus.cc:357] T 00000000000000000000000000000000 P bcc04dadc3364622b3a0e2d593b2dc60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc04dadc3364622b3a0e2d593b2dc60" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39869 } }
I20250627 13:52:50.212276 3827 raft_consensus.cc:383] T 00000000000000000000000000000000 P bcc04dadc3364622b3a0e2d593b2dc60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:50.212391 3827 raft_consensus.cc:738] T 00000000000000000000000000000000 P bcc04dadc3364622b3a0e2d593b2dc60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcc04dadc3364622b3a0e2d593b2dc60, State: Initialized, Role: FOLLOWER
I20250627 13:52:50.212946 3827 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcc04dadc3364622b3a0e2d593b2dc60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc04dadc3364622b3a0e2d593b2dc60" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39869 } }
I20250627 13:52:50.213152 3827 raft_consensus.cc:397] T 00000000000000000000000000000000 P bcc04dadc3364622b3a0e2d593b2dc60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:52:50.213251 3827 raft_consensus.cc:491] T 00000000000000000000000000000000 P bcc04dadc3364622b3a0e2d593b2dc60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:52:50.213413 3827 raft_consensus.cc:3058] T 00000000000000000000000000000000 P bcc04dadc3364622b3a0e2d593b2dc60 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:50.215519 3827 raft_consensus.cc:513] T 00000000000000000000000000000000 P bcc04dadc3364622b3a0e2d593b2dc60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc04dadc3364622b3a0e2d593b2dc60" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39869 } }
I20250627 13:52:50.215984 3827 leader_election.cc:304] T 00000000000000000000000000000000 P bcc04dadc3364622b3a0e2d593b2dc60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcc04dadc3364622b3a0e2d593b2dc60; no voters:
I20250627 13:52:50.216482 3827 leader_election.cc:290] T 00000000000000000000000000000000 P bcc04dadc3364622b3a0e2d593b2dc60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250627 13:52:50.216621 3832 raft_consensus.cc:2802] T 00000000000000000000000000000000 P bcc04dadc3364622b3a0e2d593b2dc60 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:50.217204 3832 raft_consensus.cc:695] T 00000000000000000000000000000000 P bcc04dadc3364622b3a0e2d593b2dc60 [term 1 LEADER]: Becoming Leader. State: Replica: bcc04dadc3364622b3a0e2d593b2dc60, State: Running, Role: LEADER
I20250627 13:52:50.217747 3832 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcc04dadc3364622b3a0e2d593b2dc60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc04dadc3364622b3a0e2d593b2dc60" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39869 } }
I20250627 13:52:50.218322 3827 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcc04dadc3364622b3a0e2d593b2dc60 [sys.catalog]: configured and running, proceeding with master startup.
I20250627 13:52:50.220398 3833 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc04dadc3364622b3a0e2d593b2dc60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcc04dadc3364622b3a0e2d593b2dc60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc04dadc3364622b3a0e2d593b2dc60" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39869 } } }
I20250627 13:52:50.220773 3833 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc04dadc3364622b3a0e2d593b2dc60 [sys.catalog]: This master's current role is: LEADER
I20250627 13:52:50.222721 3834 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc04dadc3364622b3a0e2d593b2dc60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcc04dadc3364622b3a0e2d593b2dc60. Latest consensus state: current_term: 1 leader_uuid: "bcc04dadc3364622b3a0e2d593b2dc60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc04dadc3364622b3a0e2d593b2dc60" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 39869 } } }
I20250627 13:52:50.223001 3834 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc04dadc3364622b3a0e2d593b2dc60 [sys.catalog]: This master's current role is: LEADER
I20250627 13:52:50.223135 3839 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250627 13:52:50.228003 3839 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250627 13:52:50.236547 3839 catalog_manager.cc:1349] Generated new cluster ID: 422c01bb145e4b8793558ffb7ab327ac
I20250627 13:52:50.236649 3839 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250627 13:52:50.269699 3839 catalog_manager.cc:1372] Generated new certificate authority record
I20250627 13:52:50.271448 3839 catalog_manager.cc:1506] Loading token signing keys...
I20250627 13:52:50.283310 3839 catalog_manager.cc:5955] T 00000000000000000000000000000000 P bcc04dadc3364622b3a0e2d593b2dc60: Generated new TSK 0
I20250627 13:52:50.284380 3839 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250627 13:52:50.301502 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.65:0
--local_ip_for_outbound_sockets=127.0.103.65
--webserver_interface=127.0.103.65
--webserver_port=0
--tserver_master_addrs=127.0.103.126:39869
--builtin_ntp_servers=127.0.103.84:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250627 13:52:50.611688 3851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:50.611985 3851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:52:50.612119 3851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:50.619887 3851 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:52:50.619993 3851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:52:50.620182 3851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.65
I20250627 13:52:50.629256 3851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.103.65
--webserver_port=0
--tserver_master_addrs=127.0.103.126:39869
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.65
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:52:50.630900 3851 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:52:50.632664 3851 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:52:50.643784 3857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:50.643810 3859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:50.643750 3856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:52:50.644747 3851 server_base.cc:1048] running on GCE node
I20250627 13:52:50.645296 3851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:52:50.646183 3851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:52:50.647389 3851 hybrid_clock.cc:648] HybridClock initialized: now 1751032370647338 us; error 29 us; skew 500 ppm
I20250627 13:52:50.647670 3851 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:52:50.649863 3851 webserver.cc:469] Webserver started at http://127.0.103.65:43099/ using document root <none> and password file <none>
I20250627 13:52:50.650584 3851 fs_manager.cc:362] Metadata directory not provided
I20250627 13:52:50.650727 3851 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:52:50.651059 3851 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:52:50.654707 3851 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f46fd51df5864af9b8493acd69db0747"
format_stamp: "Formatted at 2025-06-27 13:52:50 on dist-test-slave-1ckg"
I20250627 13:52:50.655448 3851 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f46fd51df5864af9b8493acd69db0747"
format_stamp: "Formatted at 2025-06-27 13:52:50 on dist-test-slave-1ckg"
I20250627 13:52:50.660637 3851 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20250627 13:52:50.663772 3865 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:50.664981 3851 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250627 13:52:50.665123 3851 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f46fd51df5864af9b8493acd69db0747"
format_stamp: "Formatted at 2025-06-27 13:52:50 on dist-test-slave-1ckg"
I20250627 13:52:50.665280 3851 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:52:50.682423 3851 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:52:50.683256 3851 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:52:50.683496 3851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:52:50.684321 3851 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:52:50.685822 3851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:52:50.685900 3851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:50.686034 3851 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:52:50.686093 3851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:50.715946 3851 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:35133
I20250627 13:52:50.715988 3977 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:35133 every 8 connection(s)
I20250627 13:52:50.717201 3851 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250627 13:52:50.719274 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 3851
I20250627 13:52:50.719540 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250627 13:52:50.722759 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.66:0
--local_ip_for_outbound_sockets=127.0.103.66
--webserver_interface=127.0.103.66
--webserver_port=0
--tserver_master_addrs=127.0.103.126:39869
--builtin_ntp_servers=127.0.103.84:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:52:50.730437 3978 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39869
I20250627 13:52:50.730751 3978 heartbeater.cc:461] Registering TS with master...
I20250627 13:52:50.731500 3978 heartbeater.cc:507] Master 127.0.103.126:39869 requested a full tablet report, sending...
I20250627 13:52:50.733490 3792 ts_manager.cc:194] Registered new tserver with Master: f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133)
I20250627 13:52:50.734956 3792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.65:51885
W20250627 13:52:51.020397 3982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:51.020722 3982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:52:51.020853 3982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:51.028525 3982 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:52:51.028630 3982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:52:51.028798 3982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.66
I20250627 13:52:51.037886 3982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.103.66
--webserver_port=0
--tserver_master_addrs=127.0.103.126:39869
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.66
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:52:51.039490 3982 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:52:51.041381 3982 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:52:51.052413 3988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:51.052547 3990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:51.052654 3987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:52:51.053453 3982 server_base.cc:1048] running on GCE node
I20250627 13:52:51.054080 3982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:52:51.055024 3982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:52:51.056227 3982 hybrid_clock.cc:648] HybridClock initialized: now 1751032371056167 us; error 35 us; skew 500 ppm
I20250627 13:52:51.056501 3982 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:52:51.058883 3982 webserver.cc:469] Webserver started at http://127.0.103.66:41599/ using document root <none> and password file <none>
I20250627 13:52:51.059587 3982 fs_manager.cc:362] Metadata directory not provided
I20250627 13:52:51.059711 3982 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:52:51.060037 3982 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:52:51.062582 3982 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ad800aa8621b4fc5938692202990b5ab"
format_stamp: "Formatted at 2025-06-27 13:52:51 on dist-test-slave-1ckg"
I20250627 13:52:51.063359 3982 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ad800aa8621b4fc5938692202990b5ab"
format_stamp: "Formatted at 2025-06-27 13:52:51 on dist-test-slave-1ckg"
I20250627 13:52:51.068773 3982 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20250627 13:52:51.072065 3996 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:51.073334 3982 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250627 13:52:51.073472 3982 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ad800aa8621b4fc5938692202990b5ab"
format_stamp: "Formatted at 2025-06-27 13:52:51 on dist-test-slave-1ckg"
I20250627 13:52:51.073679 3982 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:52:51.088212 3982 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:52:51.089082 3982 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:52:51.089345 3982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:52:51.090310 3982 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:52:51.091876 3982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:52:51.091948 3982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:51.092065 3982 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:52:51.092114 3982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:51.124424 3982 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:43273
I20250627 13:52:51.124464 4108 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:43273 every 8 connection(s)
I20250627 13:52:51.125801 3982 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250627 13:52:51.129994 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 3982
I20250627 13:52:51.130174 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250627 13:52:51.133404 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.67:0
--local_ip_for_outbound_sockets=127.0.103.67
--webserver_interface=127.0.103.67
--webserver_port=0
--tserver_master_addrs=127.0.103.126:39869
--builtin_ntp_servers=127.0.103.84:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:52:51.139413 4109 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39869
I20250627 13:52:51.139729 4109 heartbeater.cc:461] Registering TS with master...
I20250627 13:52:51.140654 4109 heartbeater.cc:507] Master 127.0.103.126:39869 requested a full tablet report, sending...
I20250627 13:52:51.142058 3792 ts_manager.cc:194] Registered new tserver with Master: ad800aa8621b4fc5938692202990b5ab (127.0.103.66:43273)
I20250627 13:52:51.142776 3792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.66:41129
W20250627 13:52:51.438452 4112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:51.438781 4112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:52:51.438911 4112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:51.446666 4112 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:52:51.446815 4112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:52:51.447008 4112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.67
I20250627 13:52:51.456386 4112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.103.67
--webserver_port=0
--tserver_master_addrs=127.0.103.126:39869
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.67
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:52:51.458052 4112 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:52:51.459872 4112 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:52:51.471211 4118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:51.471176 4117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:51.471246 4120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:52:51.472193 4112 server_base.cc:1048] running on GCE node
I20250627 13:52:51.472733 4112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:52:51.473584 4112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:52:51.474821 4112 hybrid_clock.cc:648] HybridClock initialized: now 1751032371474768 us; error 36 us; skew 500 ppm
I20250627 13:52:51.475083 4112 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:52:51.477166 4112 webserver.cc:469] Webserver started at http://127.0.103.67:37315/ using document root <none> and password file <none>
I20250627 13:52:51.477916 4112 fs_manager.cc:362] Metadata directory not provided
I20250627 13:52:51.478034 4112 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:52:51.478349 4112 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:52:51.480918 4112 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6"
format_stamp: "Formatted at 2025-06-27 13:52:51 on dist-test-slave-1ckg"
I20250627 13:52:51.481626 4112 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6"
format_stamp: "Formatted at 2025-06-27 13:52:51 on dist-test-slave-1ckg"
I20250627 13:52:51.486353 4112 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250627 13:52:51.489400 4126 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:51.490628 4112 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250627 13:52:51.490787 4112 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6"
format_stamp: "Formatted at 2025-06-27 13:52:51 on dist-test-slave-1ckg"
I20250627 13:52:51.490964 4112 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:52:51.508683 4112 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:52:51.509558 4112 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:52:51.509840 4112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:52:51.510710 4112 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:52:51.512176 4112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:52:51.512259 4112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:51.512378 4112 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:52:51.512434 4112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:51.545790 4112 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:44627
I20250627 13:52:51.545810 4238 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:44627 every 8 connection(s)
I20250627 13:52:51.547106 4112 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250627 13:52:51.551376 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 4112
I20250627 13:52:51.551589 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250627 13:52:51.555128 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.68:0
--local_ip_for_outbound_sockets=127.0.103.68
--webserver_interface=127.0.103.68
--webserver_port=0
--tserver_master_addrs=127.0.103.126:39869
--builtin_ntp_servers=127.0.103.84:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:52:51.561244 4239 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39869
I20250627 13:52:51.561614 4239 heartbeater.cc:461] Registering TS with master...
I20250627 13:52:51.562705 4239 heartbeater.cc:507] Master 127.0.103.126:39869 requested a full tablet report, sending...
I20250627 13:52:51.564265 3792 ts_manager.cc:194] Registered new tserver with Master: fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67:44627)
I20250627 13:52:51.565032 3792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.67:35307
I20250627 13:52:51.737715 3978 heartbeater.cc:499] Master 127.0.103.126:39869 was elected leader, sending a full tablet report...
W20250627 13:52:51.874302 4242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:51.874626 4242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:52:51.874752 4242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:51.882722 4242 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:52:51.882841 4242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:52:51.882970 4242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.68
I20250627 13:52:51.892221 4242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.103.68
--webserver_port=0
--tserver_master_addrs=127.0.103.126:39869
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.68
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:52:51.893891 4242 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:52:51.895740 4242 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:52:51.907455 4250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:51.907405 4248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:51.907502 4247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:52:51.908418 4242 server_base.cc:1048] running on GCE node
I20250627 13:52:51.909027 4242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:52:51.910041 4242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:52:51.911312 4242 hybrid_clock.cc:648] HybridClock initialized: now 1751032371911227 us; error 49 us; skew 500 ppm
I20250627 13:52:51.911636 4242 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:52:51.914160 4242 webserver.cc:469] Webserver started at http://127.0.103.68:46313/ using document root <none> and password file <none>
I20250627 13:52:51.914888 4242 fs_manager.cc:362] Metadata directory not provided
I20250627 13:52:51.915015 4242 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:52:51.915359 4242 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:52:51.917943 4242 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "36100389495d4971b10cf590bc82c42c"
format_stamp: "Formatted at 2025-06-27 13:52:51 on dist-test-slave-1ckg"
I20250627 13:52:51.918741 4242 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "36100389495d4971b10cf590bc82c42c"
format_stamp: "Formatted at 2025-06-27 13:52:51 on dist-test-slave-1ckg"
I20250627 13:52:51.924419 4242 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250627 13:52:51.927857 4256 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:51.929247 4242 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20250627 13:52:51.929410 4242 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "36100389495d4971b10cf590bc82c42c"
format_stamp: "Formatted at 2025-06-27 13:52:51 on dist-test-slave-1ckg"
I20250627 13:52:51.929584 4242 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:52:51.943110 4242 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:52:51.943907 4242 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:52:51.944170 4242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:52:51.945079 4242 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:52:51.946777 4242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:52:51.946861 4242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:51.946990 4242 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:52:51.947047 4242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:51.979792 4242 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:45503
I20250627 13:52:51.979837 4368 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:45503 every 8 connection(s)
I20250627 13:52:51.981197 4242 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250627 13:52:51.984418 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 4242
I20250627 13:52:51.984586 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250627 13:52:51.987663 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.69:0
--local_ip_for_outbound_sockets=127.0.103.69
--webserver_interface=127.0.103.69
--webserver_port=0
--tserver_master_addrs=127.0.103.126:39869
--builtin_ntp_servers=127.0.103.84:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:52:51.995008 4369 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39869
I20250627 13:52:51.995395 4369 heartbeater.cc:461] Registering TS with master...
I20250627 13:52:51.996505 4369 heartbeater.cc:507] Master 127.0.103.126:39869 requested a full tablet report, sending...
I20250627 13:52:51.998261 3792 ts_manager.cc:194] Registered new tserver with Master: 36100389495d4971b10cf590bc82c42c (127.0.103.68:45503)
I20250627 13:52:51.999070 3792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.68:42177
I20250627 13:52:52.145493 4109 heartbeater.cc:499] Master 127.0.103.126:39869 was elected leader, sending a full tablet report...
W20250627 13:52:52.309518 4372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:52.309878 4372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:52:52.309968 4372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:52.317816 4372 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:52:52.317926 4372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:52:52.318063 4372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.69
I20250627 13:52:52.327332 4372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.103.69
--webserver_port=0
--tserver_master_addrs=127.0.103.126:39869
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.69
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:52:52.328930 4372 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:52:52.330765 4372 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:52:52.342092 4378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:52.342131 4380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:52.342068 4377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:52:52.343086 4372 server_base.cc:1048] running on GCE node
I20250627 13:52:52.343732 4372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:52:52.344674 4372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:52:52.345911 4372 hybrid_clock.cc:648] HybridClock initialized: now 1751032372345839 us; error 44 us; skew 500 ppm
I20250627 13:52:52.346194 4372 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:52:52.348381 4372 webserver.cc:469] Webserver started at http://127.0.103.69:41177/ using document root <none> and password file <none>
I20250627 13:52:52.349152 4372 fs_manager.cc:362] Metadata directory not provided
I20250627 13:52:52.349278 4372 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:52:52.349606 4372 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:52:52.352100 4372 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "911714a383074e09a9b2d127ab46d2c1"
format_stamp: "Formatted at 2025-06-27 13:52:52 on dist-test-slave-1ckg"
I20250627 13:52:52.352841 4372 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "911714a383074e09a9b2d127ab46d2c1"
format_stamp: "Formatted at 2025-06-27 13:52:52 on dist-test-slave-1ckg"
I20250627 13:52:52.357965 4372 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20250627 13:52:52.361042 4386 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:52.362406 4372 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250627 13:52:52.362555 4372 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "911714a383074e09a9b2d127ab46d2c1"
format_stamp: "Formatted at 2025-06-27 13:52:52 on dist-test-slave-1ckg"
I20250627 13:52:52.362717 4372 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:52:52.381937 4372 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:52:52.382727 4372 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:52:52.382983 4372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:52:52.383893 4372 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:52:52.385363 4372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:52:52.385437 4372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:52.385556 4372 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:52:52.385609 4372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:52.417902 4372 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.69:42665
I20250627 13:52:52.417951 4498 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.69:42665 every 8 connection(s)
I20250627 13:52:52.419250 4372 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250627 13:52:52.427763 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 4372
I20250627 13:52:52.427954 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250627 13:52:52.431203 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.70:0
--local_ip_for_outbound_sockets=127.0.103.70
--webserver_interface=127.0.103.70
--webserver_port=0
--tserver_master_addrs=127.0.103.126:39869
--builtin_ntp_servers=127.0.103.84:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:52:52.431928 4499 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39869
I20250627 13:52:52.432262 4499 heartbeater.cc:461] Registering TS with master...
I20250627 13:52:52.433032 4499 heartbeater.cc:507] Master 127.0.103.126:39869 requested a full tablet report, sending...
I20250627 13:52:52.434412 3792 ts_manager.cc:194] Registered new tserver with Master: 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69:42665)
I20250627 13:52:52.435535 3792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.69:40021
I20250627 13:52:52.567793 4239 heartbeater.cc:499] Master 127.0.103.126:39869 was elected leader, sending a full tablet report...
W20250627 13:52:52.746708 4502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:52.747043 4502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:52:52.747179 4502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:52.754647 4502 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:52:52.754763 4502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:52:52.754945 4502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.70
I20250627 13:52:52.763706 4502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.103.70
--webserver_port=0
--tserver_master_addrs=127.0.103.126:39869
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.70
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:52:52.765228 4502 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:52:52.767011 4502 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:52:52.778066 4510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:52.778031 4507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:52.778119 4508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:52:52.779394 4502 server_base.cc:1048] running on GCE node
I20250627 13:52:52.779937 4502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:52:52.780809 4502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:52:52.782019 4502 hybrid_clock.cc:648] HybridClock initialized: now 1751032372781961 us; error 40 us; skew 500 ppm
I20250627 13:52:52.782284 4502 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:52:52.784406 4502 webserver.cc:469] Webserver started at http://127.0.103.70:43963/ using document root <none> and password file <none>
I20250627 13:52:52.785056 4502 fs_manager.cc:362] Metadata directory not provided
I20250627 13:52:52.785164 4502 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:52:52.785459 4502 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:52:52.787804 4502 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "bf0be937c3634bcab62f1e58402a776f"
format_stamp: "Formatted at 2025-06-27 13:52:52 on dist-test-slave-1ckg"
I20250627 13:52:52.788512 4502 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "bf0be937c3634bcab62f1e58402a776f"
format_stamp: "Formatted at 2025-06-27 13:52:52 on dist-test-slave-1ckg"
I20250627 13:52:52.793402 4502 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20250627 13:52:52.796417 4516 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:52.797576 4502 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250627 13:52:52.797765 4502 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "bf0be937c3634bcab62f1e58402a776f"
format_stamp: "Formatted at 2025-06-27 13:52:52 on dist-test-slave-1ckg"
I20250627 13:52:52.797925 4502 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:52:52.828460 4502 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:52:52.829217 4502 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:52:52.829447 4502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:52:52.830344 4502 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:52:52.831920 4502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:52:52.831987 4502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:52.832113 4502 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:52:52.832160 4502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:52.863155 4502 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.70:37215
I20250627 13:52:52.863232 4628 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.70:37215 every 8 connection(s)
I20250627 13:52:52.864444 4502 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20250627 13:52:52.869098 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 4502
I20250627 13:52:52.869271 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20250627 13:52:52.872629 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.71:0
--local_ip_for_outbound_sockets=127.0.103.71
--webserver_interface=127.0.103.71
--webserver_port=0
--tserver_master_addrs=127.0.103.126:39869
--builtin_ntp_servers=127.0.103.84:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:52:52.878608 4629 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39869
I20250627 13:52:52.878984 4629 heartbeater.cc:461] Registering TS with master...
I20250627 13:52:52.879760 4629 heartbeater.cc:507] Master 127.0.103.126:39869 requested a full tablet report, sending...
I20250627 13:52:52.881178 3792 ts_manager.cc:194] Registered new tserver with Master: bf0be937c3634bcab62f1e58402a776f (127.0.103.70:37215)
I20250627 13:52:52.881982 3792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.70:43013
I20250627 13:52:53.002521 4369 heartbeater.cc:499] Master 127.0.103.126:39869 was elected leader, sending a full tablet report...
W20250627 13:52:53.183846 4632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:53.184144 4632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:52:53.184276 4632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:53.191939 4632 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:52:53.192041 4632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:52:53.192214 4632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.71
I20250627 13:52:53.201850 4632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.103.71
--webserver_port=0
--tserver_master_addrs=127.0.103.126:39869
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.71
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:52:53.203498 4632 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:52:53.205339 4632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:52:53.216527 4637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:53.216897 4638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:52:53.216920 4640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:52:53.218116 4632 server_base.cc:1048] running on GCE node
I20250627 13:52:53.218688 4632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:52:53.219758 4632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:52:53.221004 4632 hybrid_clock.cc:648] HybridClock initialized: now 1751032373220945 us; error 35 us; skew 500 ppm
I20250627 13:52:53.221321 4632 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:52:53.223624 4632 webserver.cc:469] Webserver started at http://127.0.103.71:34299/ using document root <none> and password file <none>
I20250627 13:52:53.224335 4632 fs_manager.cc:362] Metadata directory not provided
I20250627 13:52:53.224462 4632 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:52:53.224812 4632 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:52:53.227161 4632 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "6503b59977444aaebe5ee70a0bc99154"
format_stamp: "Formatted at 2025-06-27 13:52:53 on dist-test-slave-1ckg"
I20250627 13:52:53.227886 4632 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "6503b59977444aaebe5ee70a0bc99154"
format_stamp: "Formatted at 2025-06-27 13:52:53 on dist-test-slave-1ckg"
I20250627 13:52:53.232846 4632 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250627 13:52:53.236150 4646 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:53.237367 4632 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250627 13:52:53.237504 4632 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "6503b59977444aaebe5ee70a0bc99154"
format_stamp: "Formatted at 2025-06-27 13:52:53 on dist-test-slave-1ckg"
I20250627 13:52:53.237696 4632 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:52:53.254725 4632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:52:53.255616 4632 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:52:53.255874 4632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:52:53.256793 4632 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:52:53.258420 4632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:52:53.258499 4632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:53.258638 4632 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:52:53.258693 4632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:52:53.289780 4632 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.71:45413
I20250627 13:52:53.289821 4758 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.71:45413 every 8 connection(s)
I20250627 13:52:53.291249 4632 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20250627 13:52:53.300870 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 4632
I20250627 13:52:53.301090 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20250627 13:52:53.303583 4759 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39869
I20250627 13:52:53.303936 4759 heartbeater.cc:461] Registering TS with master...
I20250627 13:52:53.304795 4759 heartbeater.cc:507] Master 127.0.103.126:39869 requested a full tablet report, sending...
I20250627 13:52:53.306147 3792 ts_manager.cc:194] Registered new tserver with Master: 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71:45413)
I20250627 13:52:53.306874 3792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.71:45979
I20250627 13:52:53.317276 413 external_mini_cluster.cc:934] 7 TS(s) registered with all masters
I20250627 13:52:53.332551 3792 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39580:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250627 13:52:53.360103 3913 tablet_service.cc:1468] Processing CreateTablet for tablet 0fdcc98da13c4cf998b1fd1b3e3f9fe5 (DEFAULT_TABLE table=TestTable [id=d060a07fcd5b4f85b0fcfc50a67aee2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:53.361629 3913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fdcc98da13c4cf998b1fd1b3e3f9fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:53.361289 4694 tablet_service.cc:1468] Processing CreateTablet for tablet 0fdcc98da13c4cf998b1fd1b3e3f9fe5 (DEFAULT_TABLE table=TestTable [id=d060a07fcd5b4f85b0fcfc50a67aee2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:53.362548 4694 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fdcc98da13c4cf998b1fd1b3e3f9fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:53.371917 4775 tablet_bootstrap.cc:492] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747: Bootstrap starting.
I20250627 13:52:53.373572 4776 tablet_bootstrap.cc:492] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154: Bootstrap starting.
I20250627 13:52:53.375249 4775 tablet_bootstrap.cc:654] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:53.376222 4775 log.cc:826] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747: Log is configured to *not* fsync() on all Append() calls
I20250627 13:52:53.376523 4776 tablet_bootstrap.cc:654] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:53.377476 4776 log.cc:826] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154: Log is configured to *not* fsync() on all Append() calls
I20250627 13:52:53.378881 4775 tablet_bootstrap.cc:492] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747: No bootstrap required, opened a new log
I20250627 13:52:53.379123 4775 ts_tablet_manager.cc:1397] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20250627 13:52:53.380445 4776 tablet_bootstrap.cc:492] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154: No bootstrap required, opened a new log
I20250627 13:52:53.380690 4776 ts_tablet_manager.cc:1397] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20250627 13:52:53.384019 4775 raft_consensus.cc:357] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:52:53.384307 4775 raft_consensus.cc:383] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:53.384431 4775 raft_consensus.cc:738] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46fd51df5864af9b8493acd69db0747, State: Initialized, Role: FOLLOWER
I20250627 13:52:53.385011 4775 consensus_queue.cc:260] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:52:53.385383 4776 raft_consensus.cc:357] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:52:53.385638 4776 raft_consensus.cc:383] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:53.385787 4776 raft_consensus.cc:738] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6503b59977444aaebe5ee70a0bc99154, State: Initialized, Role: FOLLOWER
I20250627 13:52:53.386355 4776 consensus_queue.cc:260] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:52:53.386557 4775 ts_tablet_manager.cc:1428] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250627 13:52:53.387290 4759 heartbeater.cc:499] Master 127.0.103.126:39869 was elected leader, sending a full tablet report...
I20250627 13:52:53.387982 4776 ts_tablet_manager.cc:1428] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20250627 13:52:53.438180 4499 heartbeater.cc:499] Master 127.0.103.126:39869 was elected leader, sending a full tablet report...
I20250627 13:52:53.466737 4780 raft_consensus.cc:491] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:53.467002 4780 raft_consensus.cc:513] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:52:53.468670 4780 leader_election.cc:290] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133)
W20250627 13:52:53.470532 3979 tablet.cc:2378] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:52:53.474653 3933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fdcc98da13c4cf998b1fd1b3e3f9fe5" candidate_uuid: "6503b59977444aaebe5ee70a0bc99154" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46fd51df5864af9b8493acd69db0747" is_pre_election: true
I20250627 13:52:53.475226 3933 raft_consensus.cc:2466] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6503b59977444aaebe5ee70a0bc99154 in term 0.
I20250627 13:52:53.476125 4647 leader_election.cc:304] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6503b59977444aaebe5ee70a0bc99154, f46fd51df5864af9b8493acd69db0747; no voters:
I20250627 13:52:53.476490 4780 raft_consensus.cc:2802] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:53.476627 4780 raft_consensus.cc:491] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:53.476715 4780 raft_consensus.cc:3058] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:53.478821 4780 raft_consensus.cc:513] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:52:53.479297 4780 leader_election.cc:290] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [CANDIDATE]: Term 1 election: Requested vote from peers f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133)
I20250627 13:52:53.479863 3933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fdcc98da13c4cf998b1fd1b3e3f9fe5" candidate_uuid: "6503b59977444aaebe5ee70a0bc99154" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46fd51df5864af9b8493acd69db0747"
I20250627 13:52:53.480033 3933 raft_consensus.cc:3058] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:53.482268 3933 raft_consensus.cc:2466] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6503b59977444aaebe5ee70a0bc99154 in term 1.
I20250627 13:52:53.482618 4647 leader_election.cc:304] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6503b59977444aaebe5ee70a0bc99154, f46fd51df5864af9b8493acd69db0747; no voters:
I20250627 13:52:53.482884 4780 raft_consensus.cc:2802] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:53.483287 4780 raft_consensus.cc:695] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [term 1 LEADER]: Becoming Leader. State: Replica: 6503b59977444aaebe5ee70a0bc99154, State: Running, Role: LEADER
I20250627 13:52:53.483776 4780 consensus_queue.cc:237] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:52:53.488281 3791 catalog_manager.cc:5582] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71). New cstate: current_term: 1 leader_uuid: "6503b59977444aaebe5ee70a0bc99154" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } health_report { overall_health: UNKNOWN } } }
I20250627 13:52:53.508766 413 external_mini_cluster.cc:934] 7 TS(s) registered with all masters
I20250627 13:52:53.511126 413 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f46fd51df5864af9b8493acd69db0747 to finish bootstrapping
I20250627 13:52:53.519474 413 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 6503b59977444aaebe5ee70a0bc99154 to finish bootstrapping
I20250627 13:52:53.528010 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 4242
I20250627 13:52:53.536036 4364 generic_service.cc:196] Checking for leaks (request via RPC)
W20250627 13:52:53.542903 4760 tablet.cc:2378] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:52:53.616757 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 4242
I20250627 13:52:53.632396 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 4372
I20250627 13:52:53.640031 4494 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:52:53.718776 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 4372
I20250627 13:52:53.735301 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 4502
I20250627 13:52:53.743310 4624 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:52:53.823515 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 4502
I20250627 13:52:53.839148 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 4632
I20250627 13:52:53.840180 4754 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:52:53.924347 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 4632
I20250627 13:52:55.731026 4791 raft_consensus.cc:491] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:55.731230 4791 raft_consensus.cc:513] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:52:55.732692 4791 leader_election.cc:290] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71:45413)
W20250627 13:52:55.733698 3867 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.71:45413: connect: Connection refused (error 111)
W20250627 13:52:55.735100 3867 leader_election.cc:336] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71:45413): Network error: Client connection negotiation failed: client connection to 127.0.103.71:45413: connect: Connection refused (error 111)
I20250627 13:52:55.735272 3867 leader_election.cc:304] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f46fd51df5864af9b8493acd69db0747; no voters: 6503b59977444aaebe5ee70a0bc99154
I20250627 13:52:55.735594 4791 raft_consensus.cc:2747] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:52:57.640277 4794 raft_consensus.cc:491] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:57.640470 4794 raft_consensus.cc:513] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:52:57.640976 4794 leader_election.cc:290] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71:45413)
W20250627 13:52:57.646682 3867 leader_election.cc:336] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71:45413): Network error: Client connection negotiation failed: client connection to 127.0.103.71:45413: connect: Connection refused (error 111)
I20250627 13:52:57.646843 3867 leader_election.cc:304] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f46fd51df5864af9b8493acd69db0747; no voters: 6503b59977444aaebe5ee70a0bc99154
I20250627 13:52:57.647112 4794 raft_consensus.cc:2747] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:52:57.692529 3791 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39580:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250627 13:52:57.704180 3913 tablet_service.cc:1468] Processing CreateTablet for tablet 16fed4ebf25c44069b67af98964ab075 (DEFAULT_TABLE table=rebalance_test_table_0 [id=59a8f27f17c04730859e43c3fde21ee6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:57.704807 3913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16fed4ebf25c44069b67af98964ab075. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:57.716152 4805 tablet_bootstrap.cc:492] T 16fed4ebf25c44069b67af98964ab075 P f46fd51df5864af9b8493acd69db0747: Bootstrap starting.
I20250627 13:52:57.718933 4805 tablet_bootstrap.cc:654] T 16fed4ebf25c44069b67af98964ab075 P f46fd51df5864af9b8493acd69db0747: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:57.727394 4044 tablet_service.cc:1468] Processing CreateTablet for tablet c20692dbde954a259acdb9c765978113 (DEFAULT_TABLE table=rebalance_test_table_0 [id=59a8f27f17c04730859e43c3fde21ee6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:57.728188 4042 tablet_service.cc:1468] Processing CreateTablet for tablet 16fed4ebf25c44069b67af98964ab075 (DEFAULT_TABLE table=rebalance_test_table_0 [id=59a8f27f17c04730859e43c3fde21ee6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:57.728830 4044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c20692dbde954a259acdb9c765978113. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:57.729048 4173 tablet_service.cc:1468] Processing CreateTablet for tablet 9f374c0cbcd0493597c8f30e3e63d962 (DEFAULT_TABLE table=rebalance_test_table_0 [id=59a8f27f17c04730859e43c3fde21ee6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:57.730476 4173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f374c0cbcd0493597c8f30e3e63d962. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:57.733058 4042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16fed4ebf25c44069b67af98964ab075. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:57.727198 4043 tablet_service.cc:1468] Processing CreateTablet for tablet 9f374c0cbcd0493597c8f30e3e63d962 (DEFAULT_TABLE table=rebalance_test_table_0 [id=59a8f27f17c04730859e43c3fde21ee6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:57.733477 4174 tablet_service.cc:1468] Processing CreateTablet for tablet c20692dbde954a259acdb9c765978113 (DEFAULT_TABLE table=rebalance_test_table_0 [id=59a8f27f17c04730859e43c3fde21ee6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:57.733995 4174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c20692dbde954a259acdb9c765978113. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:57.734237 4805 tablet_bootstrap.cc:492] T 16fed4ebf25c44069b67af98964ab075 P f46fd51df5864af9b8493acd69db0747: No bootstrap required, opened a new log
I20250627 13:52:57.734382 4805 ts_tablet_manager.cc:1397] T 16fed4ebf25c44069b67af98964ab075 P f46fd51df5864af9b8493acd69db0747: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.007s
I20250627 13:52:57.735409 4805 raft_consensus.cc:357] T 16fed4ebf25c44069b67af98964ab075 P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } }
I20250627 13:52:57.735657 4805 raft_consensus.cc:383] T 16fed4ebf25c44069b67af98964ab075 P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:57.735752 4805 raft_consensus.cc:738] T 16fed4ebf25c44069b67af98964ab075 P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46fd51df5864af9b8493acd69db0747, State: Initialized, Role: FOLLOWER
I20250627 13:52:57.736050 4805 consensus_queue.cc:260] T 16fed4ebf25c44069b67af98964ab075 P f46fd51df5864af9b8493acd69db0747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } }
I20250627 13:52:57.736563 4043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f374c0cbcd0493597c8f30e3e63d962. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:57.739219 4805 ts_tablet_manager.cc:1428] T 16fed4ebf25c44069b67af98964ab075 P f46fd51df5864af9b8493acd69db0747: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250627 13:52:57.746732 4808 tablet_bootstrap.cc:492] T 16fed4ebf25c44069b67af98964ab075 P ad800aa8621b4fc5938692202990b5ab: Bootstrap starting.
I20250627 13:52:57.749368 4807 tablet_bootstrap.cc:492] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Bootstrap starting.
I20250627 13:52:57.751786 4808 tablet_bootstrap.cc:654] T 16fed4ebf25c44069b67af98964ab075 P ad800aa8621b4fc5938692202990b5ab: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:57.752422 4807 tablet_bootstrap.cc:654] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:57.752779 4808 log.cc:826] T 16fed4ebf25c44069b67af98964ab075 P ad800aa8621b4fc5938692202990b5ab: Log is configured to *not* fsync() on all Append() calls
I20250627 13:52:57.753441 4807 log.cc:826] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log is configured to *not* fsync() on all Append() calls
I20250627 13:52:57.755278 4808 tablet_bootstrap.cc:492] T 16fed4ebf25c44069b67af98964ab075 P ad800aa8621b4fc5938692202990b5ab: No bootstrap required, opened a new log
I20250627 13:52:57.755519 4808 ts_tablet_manager.cc:1397] T 16fed4ebf25c44069b67af98964ab075 P ad800aa8621b4fc5938692202990b5ab: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20250627 13:52:57.755853 4807 tablet_bootstrap.cc:492] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: No bootstrap required, opened a new log
I20250627 13:52:57.756085 4807 ts_tablet_manager.cc:1397] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20250627 13:52:57.760325 4808 raft_consensus.cc:357] T 16fed4ebf25c44069b67af98964ab075 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } }
I20250627 13:52:57.760610 4808 raft_consensus.cc:383] T 16fed4ebf25c44069b67af98964ab075 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:57.760720 4808 raft_consensus.cc:738] T 16fed4ebf25c44069b67af98964ab075 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad800aa8621b4fc5938692202990b5ab, State: Initialized, Role: FOLLOWER
I20250627 13:52:57.760738 4807 raft_consensus.cc:357] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } }
I20250627 13:52:57.761011 4807 raft_consensus.cc:383] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:57.761123 4807 raft_consensus.cc:738] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb586b8cb7ab44e1a7ec5016dbd2c0a6, State: Initialized, Role: FOLLOWER
I20250627 13:52:57.761329 4808 consensus_queue.cc:260] T 16fed4ebf25c44069b67af98964ab075 P ad800aa8621b4fc5938692202990b5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } }
I20250627 13:52:57.761708 4807 consensus_queue.cc:260] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } }
I20250627 13:52:57.763041 4808 ts_tablet_manager.cc:1428] T 16fed4ebf25c44069b67af98964ab075 P ad800aa8621b4fc5938692202990b5ab: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20250627 13:52:57.763070 4807 ts_tablet_manager.cc:1428] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20250627 13:52:57.763383 4807 tablet_bootstrap.cc:492] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Bootstrap starting.
I20250627 13:52:57.763440 4808 tablet_bootstrap.cc:492] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab: Bootstrap starting.
I20250627 13:52:57.765164 4807 tablet_bootstrap.cc:654] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:57.765164 4808 tablet_bootstrap.cc:654] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:57.770653 4808 tablet_bootstrap.cc:492] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab: No bootstrap required, opened a new log
I20250627 13:52:57.770757 4807 tablet_bootstrap.cc:492] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: No bootstrap required, opened a new log
I20250627 13:52:57.770794 4808 ts_tablet_manager.cc:1397] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20250627 13:52:57.770907 4807 ts_tablet_manager.cc:1397] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20250627 13:52:57.771466 4808 raft_consensus.cc:357] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:52:57.771623 4808 raft_consensus.cc:383] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:57.771565 4807 raft_consensus.cc:357] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:52:57.771690 4808 raft_consensus.cc:738] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad800aa8621b4fc5938692202990b5ab, State: Initialized, Role: FOLLOWER
I20250627 13:52:57.771736 4807 raft_consensus.cc:383] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:57.771821 4807 raft_consensus.cc:738] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb586b8cb7ab44e1a7ec5016dbd2c0a6, State: Initialized, Role: FOLLOWER
I20250627 13:52:57.771916 4808 consensus_queue.cc:260] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:52:57.772034 4807 consensus_queue.cc:260] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:52:57.772346 4808 ts_tablet_manager.cc:1428] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250627 13:52:57.772488 4807 ts_tablet_manager.cc:1428] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250627 13:52:57.772686 4808 tablet_bootstrap.cc:492] T c20692dbde954a259acdb9c765978113 P ad800aa8621b4fc5938692202990b5ab: Bootstrap starting.
I20250627 13:52:57.774302 4808 tablet_bootstrap.cc:654] T c20692dbde954a259acdb9c765978113 P ad800aa8621b4fc5938692202990b5ab: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:57.776350 4808 tablet_bootstrap.cc:492] T c20692dbde954a259acdb9c765978113 P ad800aa8621b4fc5938692202990b5ab: No bootstrap required, opened a new log
I20250627 13:52:57.776496 4808 ts_tablet_manager.cc:1397] T c20692dbde954a259acdb9c765978113 P ad800aa8621b4fc5938692202990b5ab: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250627 13:52:57.777065 4808 raft_consensus.cc:357] T c20692dbde954a259acdb9c765978113 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } }
I20250627 13:52:57.777184 4808 raft_consensus.cc:383] T c20692dbde954a259acdb9c765978113 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:57.777289 4808 raft_consensus.cc:738] T c20692dbde954a259acdb9c765978113 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad800aa8621b4fc5938692202990b5ab, State: Initialized, Role: FOLLOWER
I20250627 13:52:57.777467 4808 consensus_queue.cc:260] T c20692dbde954a259acdb9c765978113 P ad800aa8621b4fc5938692202990b5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } }
I20250627 13:52:57.777925 4808 ts_tablet_manager.cc:1428] T c20692dbde954a259acdb9c765978113 P ad800aa8621b4fc5938692202990b5ab: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
W20250627 13:52:57.802901 4240 tablet.cc:2378] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250627 13:52:57.881863 4110 tablet.cc:2378] T 16fed4ebf25c44069b67af98964ab075 P ad800aa8621b4fc5938692202990b5ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:52:57.884593 4812 raft_consensus.cc:491] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:57.884855 4812 raft_consensus.cc:513] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:52:57.886492 4812 leader_election.cc:290] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad800aa8621b4fc5938692202990b5ab (127.0.103.66:43273)
I20250627 13:52:57.892424 4064 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f374c0cbcd0493597c8f30e3e63d962" candidate_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad800aa8621b4fc5938692202990b5ab" is_pre_election: true
I20250627 13:52:57.892901 4064 raft_consensus.cc:2466] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb586b8cb7ab44e1a7ec5016dbd2c0a6 in term 0.
I20250627 13:52:57.893549 4128 leader_election.cc:304] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ad800aa8621b4fc5938692202990b5ab, fb586b8cb7ab44e1a7ec5016dbd2c0a6; no voters:
I20250627 13:52:57.893931 4812 raft_consensus.cc:2802] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:57.894073 4812 raft_consensus.cc:491] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:57.894167 4812 raft_consensus.cc:3058] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:57.896359 4812 raft_consensus.cc:513] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:52:57.896824 4812 leader_election.cc:290] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 election: Requested vote from peers ad800aa8621b4fc5938692202990b5ab (127.0.103.66:43273)
I20250627 13:52:57.897367 4064 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f374c0cbcd0493597c8f30e3e63d962" candidate_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad800aa8621b4fc5938692202990b5ab"
I20250627 13:52:57.897533 4064 raft_consensus.cc:3058] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:57.899658 4064 raft_consensus.cc:2466] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb586b8cb7ab44e1a7ec5016dbd2c0a6 in term 1.
I20250627 13:52:57.900034 4128 leader_election.cc:304] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ad800aa8621b4fc5938692202990b5ab, fb586b8cb7ab44e1a7ec5016dbd2c0a6; no voters:
I20250627 13:52:57.900312 4812 raft_consensus.cc:2802] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:57.900748 4812 raft_consensus.cc:695] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Becoming Leader. State: Replica: fb586b8cb7ab44e1a7ec5016dbd2c0a6, State: Running, Role: LEADER
I20250627 13:52:57.901168 4812 consensus_queue.cc:237] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:52:57.905596 3792 catalog_manager.cc:5582] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } health_report { overall_health: HEALTHY } } }
I20250627 13:52:57.965194 4812 raft_consensus.cc:491] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:57.965436 4812 raft_consensus.cc:513] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } }
I20250627 13:52:57.966025 4812 leader_election.cc:290] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad800aa8621b4fc5938692202990b5ab (127.0.103.66:43273)
I20250627 13:52:57.966699 4064 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c20692dbde954a259acdb9c765978113" candidate_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad800aa8621b4fc5938692202990b5ab" is_pre_election: true
I20250627 13:52:57.966917 4064 raft_consensus.cc:2466] T c20692dbde954a259acdb9c765978113 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb586b8cb7ab44e1a7ec5016dbd2c0a6 in term 0.
I20250627 13:52:57.967356 4128 leader_election.cc:304] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ad800aa8621b4fc5938692202990b5ab, fb586b8cb7ab44e1a7ec5016dbd2c0a6; no voters:
I20250627 13:52:57.967656 4812 raft_consensus.cc:2802] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:57.967756 4812 raft_consensus.cc:491] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:57.967864 4812 raft_consensus.cc:3058] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:57.970161 4812 raft_consensus.cc:513] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } }
I20250627 13:52:57.970597 4812 leader_election.cc:290] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 election: Requested vote from peers ad800aa8621b4fc5938692202990b5ab (127.0.103.66:43273)
I20250627 13:52:57.970994 4064 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c20692dbde954a259acdb9c765978113" candidate_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad800aa8621b4fc5938692202990b5ab"
I20250627 13:52:57.971172 4064 raft_consensus.cc:3058] T c20692dbde954a259acdb9c765978113 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:57.972935 4064 raft_consensus.cc:2466] T c20692dbde954a259acdb9c765978113 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb586b8cb7ab44e1a7ec5016dbd2c0a6 in term 1.
I20250627 13:52:57.973259 4128 leader_election.cc:304] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ad800aa8621b4fc5938692202990b5ab, fb586b8cb7ab44e1a7ec5016dbd2c0a6; no voters:
I20250627 13:52:57.973505 4812 raft_consensus.cc:2802] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:57.973711 4812 raft_consensus.cc:695] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Becoming Leader. State: Replica: fb586b8cb7ab44e1a7ec5016dbd2c0a6, State: Running, Role: LEADER
I20250627 13:52:57.973949 4812 consensus_queue.cc:237] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } }
I20250627 13:52:57.976135 3792 catalog_manager.cc:5582] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } health_report { overall_health: UNKNOWN } } }
I20250627 13:52:58.020658 4811 raft_consensus.cc:491] T 16fed4ebf25c44069b67af98964ab075 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:58.020880 4811 raft_consensus.cc:513] T 16fed4ebf25c44069b67af98964ab075 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } }
I20250627 13:52:58.022490 4811 leader_election.cc:290] T 16fed4ebf25c44069b67af98964ab075 P ad800aa8621b4fc5938692202990b5ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133)
I20250627 13:52:58.028757 3933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16fed4ebf25c44069b67af98964ab075" candidate_uuid: "ad800aa8621b4fc5938692202990b5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46fd51df5864af9b8493acd69db0747" is_pre_election: true
I20250627 13:52:58.029009 3933 raft_consensus.cc:2466] T 16fed4ebf25c44069b67af98964ab075 P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad800aa8621b4fc5938692202990b5ab in term 0.
I20250627 13:52:58.029479 3997 leader_election.cc:304] T 16fed4ebf25c44069b67af98964ab075 P ad800aa8621b4fc5938692202990b5ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ad800aa8621b4fc5938692202990b5ab, f46fd51df5864af9b8493acd69db0747; no voters:
I20250627 13:52:58.029860 4811 raft_consensus.cc:2802] T 16fed4ebf25c44069b67af98964ab075 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:52:58.029989 4811 raft_consensus.cc:491] T 16fed4ebf25c44069b67af98964ab075 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:52:58.030079 4811 raft_consensus.cc:3058] T 16fed4ebf25c44069b67af98964ab075 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:58.032037 4811 raft_consensus.cc:513] T 16fed4ebf25c44069b67af98964ab075 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } }
I20250627 13:52:58.032460 4811 leader_election.cc:290] T 16fed4ebf25c44069b67af98964ab075 P ad800aa8621b4fc5938692202990b5ab [CANDIDATE]: Term 1 election: Requested vote from peers f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133)
I20250627 13:52:58.032920 3933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16fed4ebf25c44069b67af98964ab075" candidate_uuid: "ad800aa8621b4fc5938692202990b5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46fd51df5864af9b8493acd69db0747"
I20250627 13:52:58.033097 3933 raft_consensus.cc:3058] T 16fed4ebf25c44069b67af98964ab075 P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:52:58.034994 3933 raft_consensus.cc:2466] T 16fed4ebf25c44069b67af98964ab075 P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad800aa8621b4fc5938692202990b5ab in term 1.
I20250627 13:52:58.035358 3997 leader_election.cc:304] T 16fed4ebf25c44069b67af98964ab075 P ad800aa8621b4fc5938692202990b5ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ad800aa8621b4fc5938692202990b5ab, f46fd51df5864af9b8493acd69db0747; no voters:
I20250627 13:52:58.035655 4811 raft_consensus.cc:2802] T 16fed4ebf25c44069b67af98964ab075 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:52:58.036108 4811 raft_consensus.cc:695] T 16fed4ebf25c44069b67af98964ab075 P ad800aa8621b4fc5938692202990b5ab [term 1 LEADER]: Becoming Leader. State: Replica: ad800aa8621b4fc5938692202990b5ab, State: Running, Role: LEADER
I20250627 13:52:58.036595 4811 consensus_queue.cc:237] T 16fed4ebf25c44069b67af98964ab075 P ad800aa8621b4fc5938692202990b5ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } }
I20250627 13:52:58.040601 3792 catalog_manager.cc:5582] T 16fed4ebf25c44069b67af98964ab075 P ad800aa8621b4fc5938692202990b5ab reported cstate change: term changed from 0 to 1, leader changed from <none> to ad800aa8621b4fc5938692202990b5ab (127.0.103.66). New cstate: current_term: 1 leader_uuid: "ad800aa8621b4fc5938692202990b5ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } health_report { overall_health: HEALTHY } } }
W20250627 13:52:58.410844 4824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:52:58.411206 4824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:52:58.418752 4824 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20250627 13:52:58.485131 4811 consensus_queue.cc:1035] T 16fed4ebf25c44069b67af98964ab075 P ad800aa8621b4fc5938692202990b5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:58.486151 4847 consensus_queue.cc:1035] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250627 13:52:58.580858 4063 raft_consensus.cc:1273] T c20692dbde954a259acdb9c765978113 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250627 13:52:58.581580 4852 consensus_queue.cc:1035] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:52:58.665338 4854 mvcc.cc:204] Tried to move back new op lower bound from 7172228622651502592 to 7172228620182441984. Current Snapshot: MvccSnapshot[applied={T|T < 7172228622651502592}]
I20250627 13:52:58.670951 4865 mvcc.cc:204] Tried to move back new op lower bound from 7172228622651502592 to 7172228620182441984. Current Snapshot: MvccSnapshot[applied={T|T < 7172228622651502592}]
INSERT report
rows total: 2000
time total: 216.151 ms
time per row: 0.108076 ms
I20250627 13:52:59.595820 4878 raft_consensus.cc:491] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:52:59.596010 4878 raft_consensus.cc:513] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:52:59.596565 4878 leader_election.cc:290] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71:45413)
W20250627 13:52:59.598399 3867 leader_election.cc:336] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71:45413): Network error: Client connection negotiation failed: client connection to 127.0.103.71:45413: connect: Connection refused (error 111)
I20250627 13:52:59.598533 3867 leader_election.cc:304] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f46fd51df5864af9b8493acd69db0747; no voters: 6503b59977444aaebe5ee70a0bc99154
I20250627 13:52:59.598866 4878 raft_consensus.cc:2747] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20250627 13:52:59.753829 3790 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39580:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250627 13:52:59.762028 4174 tablet_service.cc:1468] Processing CreateTablet for tablet be42f2b9b1b747adb641149f70da608a (DEFAULT_TABLE table=rebalance_test_table_1 [id=70cec1ac0b6b4a82a338f38f22668b39]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:59.762223 3913 tablet_service.cc:1468] Processing CreateTablet for tablet be42f2b9b1b747adb641149f70da608a (DEFAULT_TABLE table=rebalance_test_table_1 [id=70cec1ac0b6b4a82a338f38f22668b39]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:59.762495 4174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be42f2b9b1b747adb641149f70da608a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:59.762657 3913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be42f2b9b1b747adb641149f70da608a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:59.762682 4173 tablet_service.cc:1468] Processing CreateTablet for tablet e79731c0edcc4c11a573cf46f0b2c06a (DEFAULT_TABLE table=rebalance_test_table_1 [id=70cec1ac0b6b4a82a338f38f22668b39]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:59.763015 4173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e79731c0edcc4c11a573cf46f0b2c06a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:59.763172 4044 tablet_service.cc:1468] Processing CreateTablet for tablet b2b66876e09748b49f27128fe7591be9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=70cec1ac0b6b4a82a338f38f22668b39]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:59.763542 4044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2b66876e09748b49f27128fe7591be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:59.770143 4886 tablet_bootstrap.cc:492] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Bootstrap starting.
I20250627 13:52:59.770646 3912 tablet_service.cc:1468] Processing CreateTablet for tablet e79731c0edcc4c11a573cf46f0b2c06a (DEFAULT_TABLE table=rebalance_test_table_1 [id=70cec1ac0b6b4a82a338f38f22668b39]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:59.770848 3911 tablet_service.cc:1468] Processing CreateTablet for tablet b2b66876e09748b49f27128fe7591be9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=70cec1ac0b6b4a82a338f38f22668b39]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:52:59.771081 3912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e79731c0edcc4c11a573cf46f0b2c06a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:59.771343 3911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2b66876e09748b49f27128fe7591be9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:52:59.772224 4886 tablet_bootstrap.cc:654] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:59.780339 4887 tablet_bootstrap.cc:492] T b2b66876e09748b49f27128fe7591be9 P ad800aa8621b4fc5938692202990b5ab: Bootstrap starting.
I20250627 13:52:59.782332 4885 tablet_bootstrap.cc:492] T be42f2b9b1b747adb641149f70da608a P f46fd51df5864af9b8493acd69db0747: Bootstrap starting.
I20250627 13:52:59.782506 4887 tablet_bootstrap.cc:654] T b2b66876e09748b49f27128fe7591be9 P ad800aa8621b4fc5938692202990b5ab: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:59.782745 4886 tablet_bootstrap.cc:492] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6: No bootstrap required, opened a new log
I20250627 13:52:59.782934 4886 ts_tablet_manager.cc:1397] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20250627 13:52:59.783715 4886 raft_consensus.cc:357] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:52:59.783895 4886 raft_consensus.cc:383] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:59.784035 4886 raft_consensus.cc:738] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb586b8cb7ab44e1a7ec5016dbd2c0a6, State: Initialized, Role: FOLLOWER
I20250627 13:52:59.784080 4885 tablet_bootstrap.cc:654] T be42f2b9b1b747adb641149f70da608a P f46fd51df5864af9b8493acd69db0747: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:59.784286 4886 consensus_queue.cc:260] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:52:59.784502 4887 tablet_bootstrap.cc:492] T b2b66876e09748b49f27128fe7591be9 P ad800aa8621b4fc5938692202990b5ab: No bootstrap required, opened a new log
I20250627 13:52:59.784960 4886 ts_tablet_manager.cc:1428] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20250627 13:52:59.785209 4887 ts_tablet_manager.cc:1397] T b2b66876e09748b49f27128fe7591be9 P ad800aa8621b4fc5938692202990b5ab: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20250627 13:52:59.785465 4886 tablet_bootstrap.cc:492] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Bootstrap starting.
I20250627 13:52:59.786015 4887 raft_consensus.cc:357] T b2b66876e09748b49f27128fe7591be9 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:52:59.786202 4887 raft_consensus.cc:383] T b2b66876e09748b49f27128fe7591be9 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:59.786309 4887 raft_consensus.cc:738] T b2b66876e09748b49f27128fe7591be9 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad800aa8621b4fc5938692202990b5ab, State: Initialized, Role: FOLLOWER
I20250627 13:52:59.786588 4887 consensus_queue.cc:260] T b2b66876e09748b49f27128fe7591be9 P ad800aa8621b4fc5938692202990b5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:52:59.786895 4885 tablet_bootstrap.cc:492] T be42f2b9b1b747adb641149f70da608a P f46fd51df5864af9b8493acd69db0747: No bootstrap required, opened a new log
I20250627 13:52:59.787019 4885 ts_tablet_manager.cc:1397] T be42f2b9b1b747adb641149f70da608a P f46fd51df5864af9b8493acd69db0747: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20250627 13:52:59.787042 4887 ts_tablet_manager.cc:1428] T b2b66876e09748b49f27128fe7591be9 P ad800aa8621b4fc5938692202990b5ab: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:52:59.787833 4885 raft_consensus.cc:357] T be42f2b9b1b747adb641149f70da608a P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:52:59.788031 4885 raft_consensus.cc:383] T be42f2b9b1b747adb641149f70da608a P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:59.788112 4886 tablet_bootstrap.cc:654] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:59.788132 4885 raft_consensus.cc:738] T be42f2b9b1b747adb641149f70da608a P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46fd51df5864af9b8493acd69db0747, State: Initialized, Role: FOLLOWER
I20250627 13:52:59.788372 4885 consensus_queue.cc:260] T be42f2b9b1b747adb641149f70da608a P f46fd51df5864af9b8493acd69db0747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:52:59.788905 4885 ts_tablet_manager.cc:1428] T be42f2b9b1b747adb641149f70da608a P f46fd51df5864af9b8493acd69db0747: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250627 13:52:59.789238 4885 tablet_bootstrap.cc:492] T e79731c0edcc4c11a573cf46f0b2c06a P f46fd51df5864af9b8493acd69db0747: Bootstrap starting.
I20250627 13:52:59.790640 4886 tablet_bootstrap.cc:492] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6: No bootstrap required, opened a new log
I20250627 13:52:59.790846 4886 ts_tablet_manager.cc:1397] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250627 13:52:59.790971 4885 tablet_bootstrap.cc:654] T e79731c0edcc4c11a573cf46f0b2c06a P f46fd51df5864af9b8493acd69db0747: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:59.791607 4886 raft_consensus.cc:357] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:52:59.791827 4886 raft_consensus.cc:383] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:59.791944 4886 raft_consensus.cc:738] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb586b8cb7ab44e1a7ec5016dbd2c0a6, State: Initialized, Role: FOLLOWER
I20250627 13:52:59.792228 4886 consensus_queue.cc:260] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:52:59.792783 4886 ts_tablet_manager.cc:1428] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:52:59.793869 4885 tablet_bootstrap.cc:492] T e79731c0edcc4c11a573cf46f0b2c06a P f46fd51df5864af9b8493acd69db0747: No bootstrap required, opened a new log
I20250627 13:52:59.794005 4885 ts_tablet_manager.cc:1397] T e79731c0edcc4c11a573cf46f0b2c06a P f46fd51df5864af9b8493acd69db0747: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250627 13:52:59.794585 4885 raft_consensus.cc:357] T e79731c0edcc4c11a573cf46f0b2c06a P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:52:59.794750 4885 raft_consensus.cc:383] T e79731c0edcc4c11a573cf46f0b2c06a P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:59.794823 4885 raft_consensus.cc:738] T e79731c0edcc4c11a573cf46f0b2c06a P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46fd51df5864af9b8493acd69db0747, State: Initialized, Role: FOLLOWER
I20250627 13:52:59.795046 4885 consensus_queue.cc:260] T e79731c0edcc4c11a573cf46f0b2c06a P f46fd51df5864af9b8493acd69db0747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:52:59.795511 4885 ts_tablet_manager.cc:1428] T e79731c0edcc4c11a573cf46f0b2c06a P f46fd51df5864af9b8493acd69db0747: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250627 13:52:59.795836 4885 tablet_bootstrap.cc:492] T b2b66876e09748b49f27128fe7591be9 P f46fd51df5864af9b8493acd69db0747: Bootstrap starting.
I20250627 13:52:59.797464 4885 tablet_bootstrap.cc:654] T b2b66876e09748b49f27128fe7591be9 P f46fd51df5864af9b8493acd69db0747: Neither blocks nor log segments found. Creating new log.
I20250627 13:52:59.799206 4885 tablet_bootstrap.cc:492] T b2b66876e09748b49f27128fe7591be9 P f46fd51df5864af9b8493acd69db0747: No bootstrap required, opened a new log
I20250627 13:52:59.799317 4885 ts_tablet_manager.cc:1397] T b2b66876e09748b49f27128fe7591be9 P f46fd51df5864af9b8493acd69db0747: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20250627 13:52:59.799837 4885 raft_consensus.cc:357] T b2b66876e09748b49f27128fe7591be9 P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:52:59.799957 4885 raft_consensus.cc:383] T b2b66876e09748b49f27128fe7591be9 P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:52:59.800050 4885 raft_consensus.cc:738] T b2b66876e09748b49f27128fe7591be9 P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46fd51df5864af9b8493acd69db0747, State: Initialized, Role: FOLLOWER
I20250627 13:52:59.800226 4885 consensus_queue.cc:260] T b2b66876e09748b49f27128fe7591be9 P f46fd51df5864af9b8493acd69db0747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:52:59.800571 4885 ts_tablet_manager.cc:1428] T b2b66876e09748b49f27128fe7591be9 P f46fd51df5864af9b8493acd69db0747: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250627 13:53:00.054991 4847 raft_consensus.cc:491] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:00.055253 4847 raft_consensus.cc:513] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:00.055896 4847 leader_election.cc:290] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133)
I20250627 13:53:00.060550 4878 raft_consensus.cc:491] T e79731c0edcc4c11a573cf46f0b2c06a P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:00.060736 4878 raft_consensus.cc:513] T e79731c0edcc4c11a573cf46f0b2c06a P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:00.061321 4896 raft_consensus.cc:491] T b2b66876e09748b49f27128fe7591be9 P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:00.061525 4878 leader_election.cc:290] T e79731c0edcc4c11a573cf46f0b2c06a P f46fd51df5864af9b8493acd69db0747 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67:44627)
I20250627 13:53:00.062368 4896 raft_consensus.cc:513] T b2b66876e09748b49f27128fe7591be9 P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:00.062920 3933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e79731c0edcc4c11a573cf46f0b2c06a" candidate_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46fd51df5864af9b8493acd69db0747" is_pre_election: true
I20250627 13:53:00.063143 3933 raft_consensus.cc:2466] T e79731c0edcc4c11a573cf46f0b2c06a P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb586b8cb7ab44e1a7ec5016dbd2c0a6 in term 0.
I20250627 13:53:00.063634 4127 leader_election.cc:304] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: f46fd51df5864af9b8493acd69db0747, fb586b8cb7ab44e1a7ec5016dbd2c0a6; no voters:
I20250627 13:53:00.063730 4896 leader_election.cc:290] T b2b66876e09748b49f27128fe7591be9 P f46fd51df5864af9b8493acd69db0747 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad800aa8621b4fc5938692202990b5ab (127.0.103.66:43273)
I20250627 13:53:00.063948 4847 raft_consensus.cc:2802] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:00.064033 4847 raft_consensus.cc:491] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:00.064133 4847 raft_consensus.cc:3058] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:00.067222 4847 raft_consensus.cc:513] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:00.067914 4847 leader_election.cc:290] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 election: Requested vote from peers f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133)
I20250627 13:53:00.068183 3933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e79731c0edcc4c11a573cf46f0b2c06a" candidate_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46fd51df5864af9b8493acd69db0747"
I20250627 13:53:00.068342 3933 raft_consensus.cc:3058] T e79731c0edcc4c11a573cf46f0b2c06a P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:00.069473 4194 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e79731c0edcc4c11a573cf46f0b2c06a" candidate_uuid: "f46fd51df5864af9b8493acd69db0747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" is_pre_election: true
I20250627 13:53:00.069872 4194 raft_consensus.cc:2391] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f46fd51df5864af9b8493acd69db0747 in current term 1: Already voted for candidate fb586b8cb7ab44e1a7ec5016dbd2c0a6 in this term.
I20250627 13:53:00.070729 3869 leader_election.cc:304] T e79731c0edcc4c11a573cf46f0b2c06a P f46fd51df5864af9b8493acd69db0747 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f46fd51df5864af9b8493acd69db0747; no voters: fb586b8cb7ab44e1a7ec5016dbd2c0a6
I20250627 13:53:00.071000 3933 raft_consensus.cc:2466] T e79731c0edcc4c11a573cf46f0b2c06a P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb586b8cb7ab44e1a7ec5016dbd2c0a6 in term 1.
I20250627 13:53:00.071139 4896 raft_consensus.cc:2747] T e79731c0edcc4c11a573cf46f0b2c06a P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250627 13:53:00.071352 4127 leader_election.cc:304] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: f46fd51df5864af9b8493acd69db0747, fb586b8cb7ab44e1a7ec5016dbd2c0a6; no voters:
I20250627 13:53:00.071615 4847 raft_consensus.cc:2802] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:00.072026 4847 raft_consensus.cc:695] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Becoming Leader. State: Replica: fb586b8cb7ab44e1a7ec5016dbd2c0a6, State: Running, Role: LEADER
I20250627 13:53:00.072286 4847 consensus_queue.cc:237] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:00.073319 4064 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2b66876e09748b49f27128fe7591be9" candidate_uuid: "f46fd51df5864af9b8493acd69db0747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad800aa8621b4fc5938692202990b5ab" is_pre_election: true
I20250627 13:53:00.073531 4064 raft_consensus.cc:2466] T b2b66876e09748b49f27128fe7591be9 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46fd51df5864af9b8493acd69db0747 in term 0.
I20250627 13:53:00.074085 3867 leader_election.cc:304] T b2b66876e09748b49f27128fe7591be9 P f46fd51df5864af9b8493acd69db0747 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ad800aa8621b4fc5938692202990b5ab, f46fd51df5864af9b8493acd69db0747; no voters:
I20250627 13:53:00.074348 4896 raft_consensus.cc:2802] T b2b66876e09748b49f27128fe7591be9 P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:00.074429 4896 raft_consensus.cc:491] T b2b66876e09748b49f27128fe7591be9 P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:00.074522 4896 raft_consensus.cc:3058] T b2b66876e09748b49f27128fe7591be9 P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:00.074731 3790 catalog_manager.cc:5582] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:00.076654 4896 raft_consensus.cc:513] T b2b66876e09748b49f27128fe7591be9 P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:00.077152 4896 leader_election.cc:290] T b2b66876e09748b49f27128fe7591be9 P f46fd51df5864af9b8493acd69db0747 [CANDIDATE]: Term 1 election: Requested vote from peers ad800aa8621b4fc5938692202990b5ab (127.0.103.66:43273)
I20250627 13:53:00.077600 4064 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2b66876e09748b49f27128fe7591be9" candidate_uuid: "f46fd51df5864af9b8493acd69db0747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad800aa8621b4fc5938692202990b5ab"
I20250627 13:53:00.077805 4064 raft_consensus.cc:3058] T b2b66876e09748b49f27128fe7591be9 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:00.079706 4064 raft_consensus.cc:2466] T b2b66876e09748b49f27128fe7591be9 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46fd51df5864af9b8493acd69db0747 in term 1.
I20250627 13:53:00.080093 3867 leader_election.cc:304] T b2b66876e09748b49f27128fe7591be9 P f46fd51df5864af9b8493acd69db0747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ad800aa8621b4fc5938692202990b5ab, f46fd51df5864af9b8493acd69db0747; no voters:
I20250627 13:53:00.080350 4896 raft_consensus.cc:2802] T b2b66876e09748b49f27128fe7591be9 P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:00.080518 4896 raft_consensus.cc:695] T b2b66876e09748b49f27128fe7591be9 P f46fd51df5864af9b8493acd69db0747 [term 1 LEADER]: Becoming Leader. State: Replica: f46fd51df5864af9b8493acd69db0747, State: Running, Role: LEADER
I20250627 13:53:00.080935 4896 consensus_queue.cc:237] T b2b66876e09748b49f27128fe7591be9 P f46fd51df5864af9b8493acd69db0747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:00.084311 3790 catalog_manager.cc:5582] T b2b66876e09748b49f27128fe7591be9 P f46fd51df5864af9b8493acd69db0747 reported cstate change: term changed from 0 to 1, leader changed from <none> to f46fd51df5864af9b8493acd69db0747 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "f46fd51df5864af9b8493acd69db0747" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } health_report { overall_health: HEALTHY } } }
I20250627 13:53:00.106630 4847 raft_consensus.cc:491] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:00.106806 4847 raft_consensus.cc:513] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:00.107316 4847 leader_election.cc:290] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133)
I20250627 13:53:00.107900 3933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be42f2b9b1b747adb641149f70da608a" candidate_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46fd51df5864af9b8493acd69db0747" is_pre_election: true
I20250627 13:53:00.108125 3933 raft_consensus.cc:2466] T be42f2b9b1b747adb641149f70da608a P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb586b8cb7ab44e1a7ec5016dbd2c0a6 in term 0.
I20250627 13:53:00.108527 4127 leader_election.cc:304] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: f46fd51df5864af9b8493acd69db0747, fb586b8cb7ab44e1a7ec5016dbd2c0a6; no voters:
I20250627 13:53:00.108806 4847 raft_consensus.cc:2802] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:00.108927 4847 raft_consensus.cc:491] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:00.109018 4847 raft_consensus.cc:3058] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:00.110934 4847 raft_consensus.cc:513] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:00.111361 4847 leader_election.cc:290] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 election: Requested vote from peers f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133)
I20250627 13:53:00.111855 3933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be42f2b9b1b747adb641149f70da608a" candidate_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46fd51df5864af9b8493acd69db0747"
I20250627 13:53:00.112077 3933 raft_consensus.cc:3058] T be42f2b9b1b747adb641149f70da608a P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:00.114056 3933 raft_consensus.cc:2466] T be42f2b9b1b747adb641149f70da608a P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb586b8cb7ab44e1a7ec5016dbd2c0a6 in term 1.
I20250627 13:53:00.114492 4127 leader_election.cc:304] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: f46fd51df5864af9b8493acd69db0747, fb586b8cb7ab44e1a7ec5016dbd2c0a6; no voters:
I20250627 13:53:00.114779 4847 raft_consensus.cc:2802] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:00.114917 4847 raft_consensus.cc:695] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Becoming Leader. State: Replica: fb586b8cb7ab44e1a7ec5016dbd2c0a6, State: Running, Role: LEADER
I20250627 13:53:00.115134 4847 consensus_queue.cc:237] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:00.117411 3790 catalog_manager.cc:5582] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } health_report { overall_health: UNKNOWN } } }
W20250627 13:53:00.456262 4904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:53:00.456597 4904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:53:00.464414 4904 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20250627 13:53:00.524739 4847 consensus_queue.cc:1035] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:00.588546 4896 consensus_queue.cc:1035] T b2b66876e09748b49f27128fe7591be9 P f46fd51df5864af9b8493acd69db0747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:00.697468 4847 consensus_queue.cc:1035] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 267.846 ms
time per row: 0.133923 ms
I20250627 13:53:00.877998 3790 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39580:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250627 13:53:00.886409 4044 tablet_service.cc:1468] Processing CreateTablet for tablet 4a118f20daaf45a2a1c539410dbf0d30 (DEFAULT_TABLE table=rebalance_test_table_2 [id=41b84e234b64410a9ea4c057e1c398cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:00.886540 4173 tablet_service.cc:1468] Processing CreateTablet for tablet 4a118f20daaf45a2a1c539410dbf0d30 (DEFAULT_TABLE table=rebalance_test_table_2 [id=41b84e234b64410a9ea4c057e1c398cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:00.886786 4174 tablet_service.cc:1468] Processing CreateTablet for tablet 37f6cffc2bc5456e94999b60340dfeb6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=41b84e234b64410a9ea4c057e1c398cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:00.886890 4044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a118f20daaf45a2a1c539410dbf0d30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:00.886946 4173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a118f20daaf45a2a1c539410dbf0d30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:00.887173 4174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37f6cffc2bc5456e94999b60340dfeb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:00.887632 3913 tablet_service.cc:1468] Processing CreateTablet for tablet 40b04be88e9947c5a5ca8432e3791bdd (DEFAULT_TABLE table=rebalance_test_table_2 [id=41b84e234b64410a9ea4c057e1c398cd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:00.888024 3913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40b04be88e9947c5a5ca8432e3791bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:00.894026 4043 tablet_service.cc:1468] Processing CreateTablet for tablet 37f6cffc2bc5456e94999b60340dfeb6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=41b84e234b64410a9ea4c057e1c398cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:00.894553 4043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37f6cffc2bc5456e94999b60340dfeb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:00.897534 4955 tablet_bootstrap.cc:492] T 4a118f20daaf45a2a1c539410dbf0d30 P ad800aa8621b4fc5938692202990b5ab: Bootstrap starting.
I20250627 13:53:00.894066 4042 tablet_service.cc:1468] Processing CreateTablet for tablet 40b04be88e9947c5a5ca8432e3791bdd (DEFAULT_TABLE table=rebalance_test_table_2 [id=41b84e234b64410a9ea4c057e1c398cd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:00.898198 4042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40b04be88e9947c5a5ca8432e3791bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:00.898551 4956 tablet_bootstrap.cc:492] T 37f6cffc2bc5456e94999b60340dfeb6 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Bootstrap starting.
I20250627 13:53:00.899343 4955 tablet_bootstrap.cc:654] T 4a118f20daaf45a2a1c539410dbf0d30 P ad800aa8621b4fc5938692202990b5ab: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:00.899963 4956 tablet_bootstrap.cc:654] T 37f6cffc2bc5456e94999b60340dfeb6 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:00.908042 4955 tablet_bootstrap.cc:492] T 4a118f20daaf45a2a1c539410dbf0d30 P ad800aa8621b4fc5938692202990b5ab: No bootstrap required, opened a new log
I20250627 13:53:00.908281 4955 ts_tablet_manager.cc:1397] T 4a118f20daaf45a2a1c539410dbf0d30 P ad800aa8621b4fc5938692202990b5ab: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20250627 13:53:00.908522 4956 tablet_bootstrap.cc:492] T 37f6cffc2bc5456e94999b60340dfeb6 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: No bootstrap required, opened a new log
I20250627 13:53:00.908602 4959 tablet_bootstrap.cc:492] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747: Bootstrap starting.
I20250627 13:53:00.908674 4956 ts_tablet_manager.cc:1397] T 37f6cffc2bc5456e94999b60340dfeb6 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20250627 13:53:00.909094 4955 raft_consensus.cc:357] T 4a118f20daaf45a2a1c539410dbf0d30 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:53:00.909241 4955 raft_consensus.cc:383] T 4a118f20daaf45a2a1c539410dbf0d30 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:00.909348 4955 raft_consensus.cc:738] T 4a118f20daaf45a2a1c539410dbf0d30 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad800aa8621b4fc5938692202990b5ab, State: Initialized, Role: FOLLOWER
I20250627 13:53:00.909416 4956 raft_consensus.cc:357] T 37f6cffc2bc5456e94999b60340dfeb6 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } }
I20250627 13:53:00.909621 4956 raft_consensus.cc:383] T 37f6cffc2bc5456e94999b60340dfeb6 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:00.909551 4955 consensus_queue.cc:260] T 4a118f20daaf45a2a1c539410dbf0d30 P ad800aa8621b4fc5938692202990b5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:53:00.909735 4956 raft_consensus.cc:738] T 37f6cffc2bc5456e94999b60340dfeb6 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb586b8cb7ab44e1a7ec5016dbd2c0a6, State: Initialized, Role: FOLLOWER
I20250627 13:53:00.909988 4956 consensus_queue.cc:260] T 37f6cffc2bc5456e94999b60340dfeb6 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } }
I20250627 13:53:00.910105 4955 ts_tablet_manager.cc:1428] T 4a118f20daaf45a2a1c539410dbf0d30 P ad800aa8621b4fc5938692202990b5ab: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:53:00.910360 4955 tablet_bootstrap.cc:492] T 40b04be88e9947c5a5ca8432e3791bdd P ad800aa8621b4fc5938692202990b5ab: Bootstrap starting.
I20250627 13:53:00.910915 4959 tablet_bootstrap.cc:654] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:00.911990 4955 tablet_bootstrap.cc:654] T 40b04be88e9947c5a5ca8432e3791bdd P ad800aa8621b4fc5938692202990b5ab: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:00.912077 4956 ts_tablet_manager.cc:1428] T 37f6cffc2bc5456e94999b60340dfeb6 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250627 13:53:00.912480 4956 tablet_bootstrap.cc:492] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Bootstrap starting.
I20250627 13:53:00.915993 4956 tablet_bootstrap.cc:654] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:00.919239 4959 tablet_bootstrap.cc:492] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747: No bootstrap required, opened a new log
I20250627 13:53:00.919394 4959 ts_tablet_manager.cc:1397] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20250627 13:53:00.919730 4955 tablet_bootstrap.cc:492] T 40b04be88e9947c5a5ca8432e3791bdd P ad800aa8621b4fc5938692202990b5ab: No bootstrap required, opened a new log
I20250627 13:53:00.919884 4956 tablet_bootstrap.cc:492] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: No bootstrap required, opened a new log
I20250627 13:53:00.919898 4955 ts_tablet_manager.cc:1397] T 40b04be88e9947c5a5ca8432e3791bdd P ad800aa8621b4fc5938692202990b5ab: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20250627 13:53:00.920049 4956 ts_tablet_manager.cc:1397] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20250627 13:53:00.920231 4959 raft_consensus.cc:357] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:00.920405 4959 raft_consensus.cc:383] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:00.920502 4959 raft_consensus.cc:738] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46fd51df5864af9b8493acd69db0747, State: Initialized, Role: FOLLOWER
I20250627 13:53:00.920683 4955 raft_consensus.cc:357] T 40b04be88e9947c5a5ca8432e3791bdd P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:00.920708 4956 raft_consensus.cc:357] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:53:00.920743 4959 consensus_queue.cc:260] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:00.920858 4955 raft_consensus.cc:383] T 40b04be88e9947c5a5ca8432e3791bdd P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:00.920867 4956 raft_consensus.cc:383] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:00.920934 4956 raft_consensus.cc:738] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb586b8cb7ab44e1a7ec5016dbd2c0a6, State: Initialized, Role: FOLLOWER
I20250627 13:53:00.920938 4955 raft_consensus.cc:738] T 40b04be88e9947c5a5ca8432e3791bdd P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad800aa8621b4fc5938692202990b5ab, State: Initialized, Role: FOLLOWER
I20250627 13:53:00.921236 4956 consensus_queue.cc:260] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:53:00.921590 4955 consensus_queue.cc:260] T 40b04be88e9947c5a5ca8432e3791bdd P ad800aa8621b4fc5938692202990b5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:00.921953 4956 ts_tablet_manager.cc:1428] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250627 13:53:00.922042 4959 ts_tablet_manager.cc:1428] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20250627 13:53:00.922155 4955 ts_tablet_manager.cc:1428] T 40b04be88e9947c5a5ca8432e3791bdd P ad800aa8621b4fc5938692202990b5ab: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:53:00.922453 4955 tablet_bootstrap.cc:492] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab: Bootstrap starting.
I20250627 13:53:00.924077 4955 tablet_bootstrap.cc:654] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:00.926124 4955 tablet_bootstrap.cc:492] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab: No bootstrap required, opened a new log
I20250627 13:53:00.926319 4955 ts_tablet_manager.cc:1397] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20250627 13:53:00.926874 4955 raft_consensus.cc:357] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } }
I20250627 13:53:00.927016 4955 raft_consensus.cc:383] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:00.927112 4955 raft_consensus.cc:738] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad800aa8621b4fc5938692202990b5ab, State: Initialized, Role: FOLLOWER
I20250627 13:53:00.927326 4955 consensus_queue.cc:260] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } }
I20250627 13:53:00.927757 4955 ts_tablet_manager.cc:1428] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250627 13:53:00.999740 4877 raft_consensus.cc:491] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:00.999924 4877 raft_consensus.cc:513] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } }
I20250627 13:53:01.000566 4877 leader_election.cc:290] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67:44627)
I20250627 13:53:01.006356 4194 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37f6cffc2bc5456e94999b60340dfeb6" candidate_uuid: "ad800aa8621b4fc5938692202990b5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" is_pre_election: true
I20250627 13:53:01.006562 4194 raft_consensus.cc:2466] T 37f6cffc2bc5456e94999b60340dfeb6 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad800aa8621b4fc5938692202990b5ab in term 0.
I20250627 13:53:01.006932 4000 leader_election.cc:304] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ad800aa8621b4fc5938692202990b5ab, fb586b8cb7ab44e1a7ec5016dbd2c0a6; no voters:
I20250627 13:53:01.007180 4877 raft_consensus.cc:2802] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:01.007287 4877 raft_consensus.cc:491] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:01.007351 4877 raft_consensus.cc:3058] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:01.009277 4877 raft_consensus.cc:513] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } }
I20250627 13:53:01.009727 4877 leader_election.cc:290] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [CANDIDATE]: Term 1 election: Requested vote from peers fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67:44627)
I20250627 13:53:01.010241 4194 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37f6cffc2bc5456e94999b60340dfeb6" candidate_uuid: "ad800aa8621b4fc5938692202990b5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6"
I20250627 13:53:01.010416 4194 raft_consensus.cc:3058] T 37f6cffc2bc5456e94999b60340dfeb6 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:01.012365 4194 raft_consensus.cc:2466] T 37f6cffc2bc5456e94999b60340dfeb6 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad800aa8621b4fc5938692202990b5ab in term 1.
I20250627 13:53:01.012720 4000 leader_election.cc:304] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ad800aa8621b4fc5938692202990b5ab, fb586b8cb7ab44e1a7ec5016dbd2c0a6; no voters:
I20250627 13:53:01.012990 4877 raft_consensus.cc:2802] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:01.013401 4877 raft_consensus.cc:695] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [term 1 LEADER]: Becoming Leader. State: Replica: ad800aa8621b4fc5938692202990b5ab, State: Running, Role: LEADER
I20250627 13:53:01.013797 4877 consensus_queue.cc:237] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } }
I20250627 13:53:01.016495 3783 catalog_manager.cc:5582] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab reported cstate change: term changed from 0 to 1, leader changed from <none> to ad800aa8621b4fc5938692202990b5ab (127.0.103.66). New cstate: current_term: 1 leader_uuid: "ad800aa8621b4fc5938692202990b5ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } health_report { overall_health: HEALTHY } } }
I20250627 13:53:01.094251 4934 raft_consensus.cc:491] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:01.094446 4934 raft_consensus.cc:513] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:01.095045 4934 leader_election.cc:290] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad800aa8621b4fc5938692202990b5ab (127.0.103.66:43273)
I20250627 13:53:01.095682 4063 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40b04be88e9947c5a5ca8432e3791bdd" candidate_uuid: "f46fd51df5864af9b8493acd69db0747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad800aa8621b4fc5938692202990b5ab" is_pre_election: true
I20250627 13:53:01.095932 4063 raft_consensus.cc:2466] T 40b04be88e9947c5a5ca8432e3791bdd P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46fd51df5864af9b8493acd69db0747 in term 0.
I20250627 13:53:01.096385 3867 leader_election.cc:304] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ad800aa8621b4fc5938692202990b5ab, f46fd51df5864af9b8493acd69db0747; no voters:
I20250627 13:53:01.096680 4934 raft_consensus.cc:2802] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:01.096781 4934 raft_consensus.cc:491] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:01.096843 4934 raft_consensus.cc:3058] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:01.098816 4934 raft_consensus.cc:513] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:01.099201 4934 leader_election.cc:290] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 [CANDIDATE]: Term 1 election: Requested vote from peers ad800aa8621b4fc5938692202990b5ab (127.0.103.66:43273)
I20250627 13:53:01.099822 4063 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40b04be88e9947c5a5ca8432e3791bdd" candidate_uuid: "f46fd51df5864af9b8493acd69db0747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad800aa8621b4fc5938692202990b5ab"
I20250627 13:53:01.100003 4063 raft_consensus.cc:3058] T 40b04be88e9947c5a5ca8432e3791bdd P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:01.102001 4063 raft_consensus.cc:2466] T 40b04be88e9947c5a5ca8432e3791bdd P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f46fd51df5864af9b8493acd69db0747 in term 1.
I20250627 13:53:01.102421 3867 leader_election.cc:304] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ad800aa8621b4fc5938692202990b5ab, f46fd51df5864af9b8493acd69db0747; no voters:
I20250627 13:53:01.102710 4934 raft_consensus.cc:2802] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:01.102841 4934 raft_consensus.cc:695] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 [term 1 LEADER]: Becoming Leader. State: Replica: f46fd51df5864af9b8493acd69db0747, State: Running, Role: LEADER
I20250627 13:53:01.103127 4934 consensus_queue.cc:237] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:01.105763 3783 catalog_manager.cc:5582] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 reported cstate change: term changed from 0 to 1, leader changed from <none> to f46fd51df5864af9b8493acd69db0747 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "f46fd51df5864af9b8493acd69db0747" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } health_report { overall_health: HEALTHY } } }
I20250627 13:53:01.169608 4847 raft_consensus.cc:491] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:01.169903 4847 raft_consensus.cc:513] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:53:01.170439 4847 leader_election.cc:290] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad800aa8621b4fc5938692202990b5ab (127.0.103.66:43273)
I20250627 13:53:01.171049 4063 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a118f20daaf45a2a1c539410dbf0d30" candidate_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad800aa8621b4fc5938692202990b5ab" is_pre_election: true
I20250627 13:53:01.171236 4063 raft_consensus.cc:2466] T 4a118f20daaf45a2a1c539410dbf0d30 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb586b8cb7ab44e1a7ec5016dbd2c0a6 in term 0.
I20250627 13:53:01.171629 4128 leader_election.cc:304] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ad800aa8621b4fc5938692202990b5ab, fb586b8cb7ab44e1a7ec5016dbd2c0a6; no voters:
I20250627 13:53:01.171902 4847 raft_consensus.cc:2802] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:01.171991 4847 raft_consensus.cc:491] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:01.172092 4847 raft_consensus.cc:3058] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:01.173933 4847 raft_consensus.cc:513] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:53:01.174328 4847 leader_election.cc:290] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 election: Requested vote from peers ad800aa8621b4fc5938692202990b5ab (127.0.103.66:43273)
I20250627 13:53:01.174834 4063 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a118f20daaf45a2a1c539410dbf0d30" candidate_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad800aa8621b4fc5938692202990b5ab"
I20250627 13:53:01.175017 4063 raft_consensus.cc:3058] T 4a118f20daaf45a2a1c539410dbf0d30 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:01.176838 4063 raft_consensus.cc:2466] T 4a118f20daaf45a2a1c539410dbf0d30 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb586b8cb7ab44e1a7ec5016dbd2c0a6 in term 1.
I20250627 13:53:01.177237 4128 leader_election.cc:304] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ad800aa8621b4fc5938692202990b5ab, fb586b8cb7ab44e1a7ec5016dbd2c0a6; no voters:
I20250627 13:53:01.177493 4847 raft_consensus.cc:2802] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:01.177719 4847 raft_consensus.cc:695] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Becoming Leader. State: Replica: fb586b8cb7ab44e1a7ec5016dbd2c0a6, State: Running, Role: LEADER
I20250627 13:53:01.178004 4847 consensus_queue.cc:237] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:53:01.180315 3783 catalog_manager.cc:5582] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } health_report { overall_health: HEALTHY } } }
I20250627 13:53:01.421627 4877 consensus_queue.cc:1035] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:01.494073 4934 consensus_queue.cc:1035] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250627 13:53:01.514494 4969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:53:01.514904 4969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:53:01.522609 4969 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20250627 13:53:01.684062 4063 raft_consensus.cc:1273] T 4a118f20daaf45a2a1c539410dbf0d30 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250627 13:53:01.684841 4847 consensus_queue.cc:1035] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 204.946 ms
time per row: 0.102473 ms
I20250627 13:53:01.871131 3786 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39580:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250627 13:53:01.879817 3913 tablet_service.cc:1468] Processing CreateTablet for tablet 1cd6f75e462546c6bb05ef14af3498fa (DEFAULT_TABLE table=rebalance_test_table_3 [id=d219e402e8284b468ef8ca703004dd5f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:01.880272 3913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cd6f75e462546c6bb05ef14af3498fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:01.880476 3911 tablet_service.cc:1468] Processing CreateTablet for tablet 3e2b11c6788e4ccdbdde60769fee1aae (DEFAULT_TABLE table=rebalance_test_table_3 [id=d219e402e8284b468ef8ca703004dd5f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:01.880489 4043 tablet_service.cc:1468] Processing CreateTablet for tablet 76518bc59dfe443abd8e97539d1bcb8a (DEFAULT_TABLE table=rebalance_test_table_3 [id=d219e402e8284b468ef8ca703004dd5f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:01.880856 3911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e2b11c6788e4ccdbdde60769fee1aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:01.880862 4043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76518bc59dfe443abd8e97539d1bcb8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:01.885356 3912 tablet_service.cc:1468] Processing CreateTablet for tablet 76518bc59dfe443abd8e97539d1bcb8a (DEFAULT_TABLE table=rebalance_test_table_3 [id=d219e402e8284b468ef8ca703004dd5f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:01.885771 3912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76518bc59dfe443abd8e97539d1bcb8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:01.885947 4174 tablet_service.cc:1468] Processing CreateTablet for tablet 1cd6f75e462546c6bb05ef14af3498fa (DEFAULT_TABLE table=rebalance_test_table_3 [id=d219e402e8284b468ef8ca703004dd5f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:01.886049 4173 tablet_service.cc:1468] Processing CreateTablet for tablet 3e2b11c6788e4ccdbdde60769fee1aae (DEFAULT_TABLE table=rebalance_test_table_3 [id=d219e402e8284b468ef8ca703004dd5f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:01.886350 4174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cd6f75e462546c6bb05ef14af3498fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:01.886576 4173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e2b11c6788e4ccdbdde60769fee1aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:01.891338 5024 tablet_bootstrap.cc:492] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab: Bootstrap starting.
I20250627 13:53:01.892277 5023 tablet_bootstrap.cc:492] T 1cd6f75e462546c6bb05ef14af3498fa P f46fd51df5864af9b8493acd69db0747: Bootstrap starting.
I20250627 13:53:01.893311 5024 tablet_bootstrap.cc:654] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:01.893553 5025 tablet_bootstrap.cc:492] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Bootstrap starting.
I20250627 13:53:01.894106 5023 tablet_bootstrap.cc:654] T 1cd6f75e462546c6bb05ef14af3498fa P f46fd51df5864af9b8493acd69db0747: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:01.896788 5025 tablet_bootstrap.cc:654] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:01.898409 5024 tablet_bootstrap.cc:492] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab: No bootstrap required, opened a new log
I20250627 13:53:01.898566 5024 ts_tablet_manager.cc:1397] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20250627 13:53:01.898810 5025 tablet_bootstrap.cc:492] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6: No bootstrap required, opened a new log
I20250627 13:53:01.898906 5023 tablet_bootstrap.cc:492] T 1cd6f75e462546c6bb05ef14af3498fa P f46fd51df5864af9b8493acd69db0747: No bootstrap required, opened a new log
I20250627 13:53:01.898931 5025 ts_tablet_manager.cc:1397] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20250627 13:53:01.899052 5023 ts_tablet_manager.cc:1397] T 1cd6f75e462546c6bb05ef14af3498fa P f46fd51df5864af9b8493acd69db0747: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20250627 13:53:01.899413 5024 raft_consensus.cc:357] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:01.899634 5024 raft_consensus.cc:383] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:01.899657 5025 raft_consensus.cc:357] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:53:01.899758 5024 raft_consensus.cc:738] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad800aa8621b4fc5938692202990b5ab, State: Initialized, Role: FOLLOWER
I20250627 13:53:01.899832 5025 raft_consensus.cc:383] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:01.899828 5023 raft_consensus.cc:357] T 1cd6f75e462546c6bb05ef14af3498fa P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:53:01.899943 5025 raft_consensus.cc:738] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb586b8cb7ab44e1a7ec5016dbd2c0a6, State: Initialized, Role: FOLLOWER
I20250627 13:53:01.900043 5023 raft_consensus.cc:383] T 1cd6f75e462546c6bb05ef14af3498fa P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:01.900146 5023 raft_consensus.cc:738] T 1cd6f75e462546c6bb05ef14af3498fa P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46fd51df5864af9b8493acd69db0747, State: Initialized, Role: FOLLOWER
I20250627 13:53:01.900104 5024 consensus_queue.cc:260] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:01.900276 5025 consensus_queue.cc:260] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:53:01.900396 5023 consensus_queue.cc:260] T 1cd6f75e462546c6bb05ef14af3498fa P f46fd51df5864af9b8493acd69db0747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:53:01.900842 5025 ts_tablet_manager.cc:1428] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250627 13:53:01.900936 5023 ts_tablet_manager.cc:1428] T 1cd6f75e462546c6bb05ef14af3498fa P f46fd51df5864af9b8493acd69db0747: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250627 13:53:01.901260 5023 tablet_bootstrap.cc:492] T 3e2b11c6788e4ccdbdde60769fee1aae P f46fd51df5864af9b8493acd69db0747: Bootstrap starting.
I20250627 13:53:01.901829 5025 tablet_bootstrap.cc:492] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Bootstrap starting.
I20250627 13:53:01.902714 5024 ts_tablet_manager.cc:1428] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20250627 13:53:01.903254 5023 tablet_bootstrap.cc:654] T 3e2b11c6788e4ccdbdde60769fee1aae P f46fd51df5864af9b8493acd69db0747: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:01.903559 5025 tablet_bootstrap.cc:654] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:01.905537 5023 tablet_bootstrap.cc:492] T 3e2b11c6788e4ccdbdde60769fee1aae P f46fd51df5864af9b8493acd69db0747: No bootstrap required, opened a new log
I20250627 13:53:01.905758 5023 ts_tablet_manager.cc:1397] T 3e2b11c6788e4ccdbdde60769fee1aae P f46fd51df5864af9b8493acd69db0747: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250627 13:53:01.906145 5025 tablet_bootstrap.cc:492] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6: No bootstrap required, opened a new log
I20250627 13:53:01.906287 5025 ts_tablet_manager.cc:1397] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20250627 13:53:01.906553 5023 raft_consensus.cc:357] T 3e2b11c6788e4ccdbdde60769fee1aae P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:01.906751 5023 raft_consensus.cc:383] T 3e2b11c6788e4ccdbdde60769fee1aae P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:01.906877 5023 raft_consensus.cc:738] T 3e2b11c6788e4ccdbdde60769fee1aae P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46fd51df5864af9b8493acd69db0747, State: Initialized, Role: FOLLOWER
I20250627 13:53:01.906895 5025 raft_consensus.cc:357] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:01.907114 5025 raft_consensus.cc:383] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:01.907217 5025 raft_consensus.cc:738] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb586b8cb7ab44e1a7ec5016dbd2c0a6, State: Initialized, Role: FOLLOWER
I20250627 13:53:01.907158 5023 consensus_queue.cc:260] T 3e2b11c6788e4ccdbdde60769fee1aae P f46fd51df5864af9b8493acd69db0747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:01.907508 5025 consensus_queue.cc:260] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:01.907776 5023 ts_tablet_manager.cc:1428] T 3e2b11c6788e4ccdbdde60769fee1aae P f46fd51df5864af9b8493acd69db0747: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:53:01.908005 5025 ts_tablet_manager.cc:1428] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250627 13:53:01.908079 5023 tablet_bootstrap.cc:492] T 76518bc59dfe443abd8e97539d1bcb8a P f46fd51df5864af9b8493acd69db0747: Bootstrap starting.
I20250627 13:53:01.909868 5023 tablet_bootstrap.cc:654] T 76518bc59dfe443abd8e97539d1bcb8a P f46fd51df5864af9b8493acd69db0747: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:01.911711 5023 tablet_bootstrap.cc:492] T 76518bc59dfe443abd8e97539d1bcb8a P f46fd51df5864af9b8493acd69db0747: No bootstrap required, opened a new log
I20250627 13:53:01.911875 5023 ts_tablet_manager.cc:1397] T 76518bc59dfe443abd8e97539d1bcb8a P f46fd51df5864af9b8493acd69db0747: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250627 13:53:01.912462 5023 raft_consensus.cc:357] T 76518bc59dfe443abd8e97539d1bcb8a P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:01.912590 5023 raft_consensus.cc:383] T 76518bc59dfe443abd8e97539d1bcb8a P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:01.912653 5023 raft_consensus.cc:738] T 76518bc59dfe443abd8e97539d1bcb8a P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46fd51df5864af9b8493acd69db0747, State: Initialized, Role: FOLLOWER
I20250627 13:53:01.912812 5023 consensus_queue.cc:260] T 76518bc59dfe443abd8e97539d1bcb8a P f46fd51df5864af9b8493acd69db0747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:01.913234 5023 ts_tablet_manager.cc:1428] T 76518bc59dfe443abd8e97539d1bcb8a P f46fd51df5864af9b8493acd69db0747: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250627 13:53:01.916325 4965 raft_consensus.cc:491] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:01.916497 4965 raft_consensus.cc:513] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:01.917073 4965 leader_election.cc:290] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133)
I20250627 13:53:01.917472 3932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76518bc59dfe443abd8e97539d1bcb8a" candidate_uuid: "ad800aa8621b4fc5938692202990b5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46fd51df5864af9b8493acd69db0747" is_pre_election: true
I20250627 13:53:01.917655 3932 raft_consensus.cc:2466] T 76518bc59dfe443abd8e97539d1bcb8a P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad800aa8621b4fc5938692202990b5ab in term 0.
I20250627 13:53:01.918172 3997 leader_election.cc:304] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ad800aa8621b4fc5938692202990b5ab, f46fd51df5864af9b8493acd69db0747; no voters:
I20250627 13:53:01.918443 4965 raft_consensus.cc:2802] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:01.918557 4965 raft_consensus.cc:491] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:01.918648 4965 raft_consensus.cc:3058] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:01.921043 4965 raft_consensus.cc:513] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:01.921523 4965 leader_election.cc:290] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab [CANDIDATE]: Term 1 election: Requested vote from peers f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133)
I20250627 13:53:01.922081 3932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76518bc59dfe443abd8e97539d1bcb8a" candidate_uuid: "ad800aa8621b4fc5938692202990b5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46fd51df5864af9b8493acd69db0747"
I20250627 13:53:01.922252 3932 raft_consensus.cc:3058] T 76518bc59dfe443abd8e97539d1bcb8a P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:01.924311 3932 raft_consensus.cc:2466] T 76518bc59dfe443abd8e97539d1bcb8a P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad800aa8621b4fc5938692202990b5ab in term 1.
I20250627 13:53:01.924821 3997 leader_election.cc:304] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ad800aa8621b4fc5938692202990b5ab, f46fd51df5864af9b8493acd69db0747; no voters:
I20250627 13:53:01.925137 4965 raft_consensus.cc:2802] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:01.925279 4965 raft_consensus.cc:695] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab [term 1 LEADER]: Becoming Leader. State: Replica: ad800aa8621b4fc5938692202990b5ab, State: Running, Role: LEADER
I20250627 13:53:01.925549 4965 consensus_queue.cc:237] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:01.928081 3789 catalog_manager.cc:5582] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab reported cstate change: term changed from 0 to 1, leader changed from <none> to ad800aa8621b4fc5938692202990b5ab (127.0.103.66). New cstate: current_term: 1 leader_uuid: "ad800aa8621b4fc5938692202990b5ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:01.934964 5013 raft_consensus.cc:491] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:01.935122 5013 raft_consensus.cc:513] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:01.935573 5013 leader_election.cc:290] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133)
I20250627 13:53:01.936265 3933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e2b11c6788e4ccdbdde60769fee1aae" candidate_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46fd51df5864af9b8493acd69db0747" is_pre_election: true
I20250627 13:53:01.936482 3933 raft_consensus.cc:2466] T 3e2b11c6788e4ccdbdde60769fee1aae P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb586b8cb7ab44e1a7ec5016dbd2c0a6 in term 0.
I20250627 13:53:01.936940 4127 leader_election.cc:304] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: f46fd51df5864af9b8493acd69db0747, fb586b8cb7ab44e1a7ec5016dbd2c0a6; no voters:
I20250627 13:53:01.937222 5013 raft_consensus.cc:2802] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:01.937325 5013 raft_consensus.cc:491] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:01.937415 5013 raft_consensus.cc:3058] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:01.939388 5013 raft_consensus.cc:513] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:01.939816 5013 leader_election.cc:290] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 election: Requested vote from peers f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133)
I20250627 13:53:01.940349 3933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e2b11c6788e4ccdbdde60769fee1aae" candidate_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46fd51df5864af9b8493acd69db0747"
I20250627 13:53:01.940517 3933 raft_consensus.cc:3058] T 3e2b11c6788e4ccdbdde60769fee1aae P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:01.942516 3933 raft_consensus.cc:2466] T 3e2b11c6788e4ccdbdde60769fee1aae P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb586b8cb7ab44e1a7ec5016dbd2c0a6 in term 1.
I20250627 13:53:01.942929 4127 leader_election.cc:304] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: f46fd51df5864af9b8493acd69db0747, fb586b8cb7ab44e1a7ec5016dbd2c0a6; no voters:
I20250627 13:53:01.943225 5013 raft_consensus.cc:2802] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:01.943358 5013 raft_consensus.cc:695] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Becoming Leader. State: Replica: fb586b8cb7ab44e1a7ec5016dbd2c0a6, State: Running, Role: LEADER
I20250627 13:53:01.943605 5013 consensus_queue.cc:237] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:01.946097 3789 catalog_manager.cc:5582] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:02.112704 5013 raft_consensus.cc:491] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:02.112923 5013 raft_consensus.cc:513] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:53:02.113370 5013 leader_election.cc:290] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133)
I20250627 13:53:02.113977 3933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1cd6f75e462546c6bb05ef14af3498fa" candidate_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46fd51df5864af9b8493acd69db0747" is_pre_election: true
I20250627 13:53:02.114202 3933 raft_consensus.cc:2466] T 1cd6f75e462546c6bb05ef14af3498fa P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb586b8cb7ab44e1a7ec5016dbd2c0a6 in term 0.
I20250627 13:53:02.114602 4127 leader_election.cc:304] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: f46fd51df5864af9b8493acd69db0747, fb586b8cb7ab44e1a7ec5016dbd2c0a6; no voters:
I20250627 13:53:02.114904 5013 raft_consensus.cc:2802] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:02.114992 5013 raft_consensus.cc:491] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:02.115109 5013 raft_consensus.cc:3058] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:02.117067 5013 raft_consensus.cc:513] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:53:02.117527 5013 leader_election.cc:290] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 election: Requested vote from peers f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133)
I20250627 13:53:02.118034 3933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1cd6f75e462546c6bb05ef14af3498fa" candidate_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46fd51df5864af9b8493acd69db0747"
I20250627 13:53:02.118240 3933 raft_consensus.cc:3058] T 1cd6f75e462546c6bb05ef14af3498fa P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:02.120184 3933 raft_consensus.cc:2466] T 1cd6f75e462546c6bb05ef14af3498fa P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb586b8cb7ab44e1a7ec5016dbd2c0a6 in term 1.
I20250627 13:53:02.120574 4127 leader_election.cc:304] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: f46fd51df5864af9b8493acd69db0747, fb586b8cb7ab44e1a7ec5016dbd2c0a6; no voters:
I20250627 13:53:02.120846 5013 raft_consensus.cc:2802] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:02.121016 5013 raft_consensus.cc:695] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Becoming Leader. State: Replica: fb586b8cb7ab44e1a7ec5016dbd2c0a6, State: Running, Role: LEADER
I20250627 13:53:02.121261 5013 consensus_queue.cc:237] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:53:02.123991 3789 catalog_manager.cc:5582] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } health_report { overall_health: HEALTHY } } }
I20250627 13:53:02.445808 4965 consensus_queue.cc:1035] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250627 13:53:02.463521 5035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:53:02.463882 5035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:53:02.471936 5035 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250627 13:53:02.496616 5013 consensus_queue.cc:1035] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20250627 13:53:02.529106 4847 consensus_queue.cc:1035] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 189.992 ms
time per row: 0.094996 ms
I20250627 13:53:03.743152 5011 raft_consensus.cc:491] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:03.743335 5011 raft_consensus.cc:513] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:03.743849 5011 leader_election.cc:290] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71:45413)
W20250627 13:53:03.745043 3867 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.71:45413: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20250627 13:53:03.746610 3867 leader_election.cc:336] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71:45413): Network error: Client connection negotiation failed: client connection to 127.0.103.71:45413: connect: Connection refused (error 111)
I20250627 13:53:03.746794 3867 leader_election.cc:304] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f46fd51df5864af9b8493acd69db0747; no voters: 6503b59977444aaebe5ee70a0bc99154
I20250627 13:53:03.747103 5011 raft_consensus.cc:2747] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20250627 13:53:03.794662 3789 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39580:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250627 13:53:03.804116 4043 tablet_service.cc:1468] Processing CreateTablet for tablet 978e20547d644d3988d6da29235903aa (DEFAULT_TABLE table=rebalance_test_table_4 [id=b5d412773a2343d1919352385a3629db]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:03.804260 4173 tablet_service.cc:1468] Processing CreateTablet for tablet 978e20547d644d3988d6da29235903aa (DEFAULT_TABLE table=rebalance_test_table_4 [id=b5d412773a2343d1919352385a3629db]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:03.804397 4042 tablet_service.cc:1468] Processing CreateTablet for tablet 8fe13ad746124b1d82cfb8ce691828a4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=b5d412773a2343d1919352385a3629db]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:03.804613 4043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 978e20547d644d3988d6da29235903aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:03.804720 4173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 978e20547d644d3988d6da29235903aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:03.805088 4042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fe13ad746124b1d82cfb8ce691828a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:03.805732 4174 tablet_service.cc:1468] Processing CreateTablet for tablet 8fe13ad746124b1d82cfb8ce691828a4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=b5d412773a2343d1919352385a3629db]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:03.806097 4174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fe13ad746124b1d82cfb8ce691828a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:03.810312 4044 tablet_service.cc:1468] Processing CreateTablet for tablet a01527b9f7c14748837f5ad4dbaa9e9d (DEFAULT_TABLE table=rebalance_test_table_4 [id=b5d412773a2343d1919352385a3629db]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:03.810711 4044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a01527b9f7c14748837f5ad4dbaa9e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:03.810662 3912 tablet_service.cc:1468] Processing CreateTablet for tablet a01527b9f7c14748837f5ad4dbaa9e9d (DEFAULT_TABLE table=rebalance_test_table_4 [id=b5d412773a2343d1919352385a3629db]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:03.811092 3912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a01527b9f7c14748837f5ad4dbaa9e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:03.820479 5096 tablet_bootstrap.cc:492] T 8fe13ad746124b1d82cfb8ce691828a4 P ad800aa8621b4fc5938692202990b5ab: Bootstrap starting.
I20250627 13:53:03.821363 5098 tablet_bootstrap.cc:492] T a01527b9f7c14748837f5ad4dbaa9e9d P f46fd51df5864af9b8493acd69db0747: Bootstrap starting.
I20250627 13:53:03.822284 5097 tablet_bootstrap.cc:492] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Bootstrap starting.
I20250627 13:53:03.823112 5096 tablet_bootstrap.cc:654] T 8fe13ad746124b1d82cfb8ce691828a4 P ad800aa8621b4fc5938692202990b5ab: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:03.823200 5098 tablet_bootstrap.cc:654] T a01527b9f7c14748837f5ad4dbaa9e9d P f46fd51df5864af9b8493acd69db0747: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:03.823886 5097 tablet_bootstrap.cc:654] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:03.825168 5096 tablet_bootstrap.cc:492] T 8fe13ad746124b1d82cfb8ce691828a4 P ad800aa8621b4fc5938692202990b5ab: No bootstrap required, opened a new log
I20250627 13:53:03.825309 5096 ts_tablet_manager.cc:1397] T 8fe13ad746124b1d82cfb8ce691828a4 P ad800aa8621b4fc5938692202990b5ab: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250627 13:53:03.825699 5097 tablet_bootstrap.cc:492] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: No bootstrap required, opened a new log
I20250627 13:53:03.825700 5098 tablet_bootstrap.cc:492] T a01527b9f7c14748837f5ad4dbaa9e9d P f46fd51df5864af9b8493acd69db0747: No bootstrap required, opened a new log
I20250627 13:53:03.825838 5098 ts_tablet_manager.cc:1397] T a01527b9f7c14748837f5ad4dbaa9e9d P f46fd51df5864af9b8493acd69db0747: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20250627 13:53:03.825839 5097 ts_tablet_manager.cc:1397] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20250627 13:53:03.826085 5096 raft_consensus.cc:357] T 8fe13ad746124b1d82cfb8ce691828a4 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:53:03.826234 5096 raft_consensus.cc:383] T 8fe13ad746124b1d82cfb8ce691828a4 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:03.826324 5096 raft_consensus.cc:738] T 8fe13ad746124b1d82cfb8ce691828a4 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad800aa8621b4fc5938692202990b5ab, State: Initialized, Role: FOLLOWER
I20250627 13:53:03.826508 5096 consensus_queue.cc:260] T 8fe13ad746124b1d82cfb8ce691828a4 P ad800aa8621b4fc5938692202990b5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:53:03.826545 5098 raft_consensus.cc:357] T a01527b9f7c14748837f5ad4dbaa9e9d P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } }
I20250627 13:53:03.826545 5097 raft_consensus.cc:357] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:53:03.826714 5097 raft_consensus.cc:383] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:03.826714 5098 raft_consensus.cc:383] T a01527b9f7c14748837f5ad4dbaa9e9d P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:03.826794 5098 raft_consensus.cc:738] T a01527b9f7c14748837f5ad4dbaa9e9d P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46fd51df5864af9b8493acd69db0747, State: Initialized, Role: FOLLOWER
I20250627 13:53:03.826794 5097 raft_consensus.cc:738] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb586b8cb7ab44e1a7ec5016dbd2c0a6, State: Initialized, Role: FOLLOWER
I20250627 13:53:03.826977 5096 ts_tablet_manager.cc:1428] T 8fe13ad746124b1d82cfb8ce691828a4 P ad800aa8621b4fc5938692202990b5ab: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:53:03.826974 5098 consensus_queue.cc:260] T a01527b9f7c14748837f5ad4dbaa9e9d P f46fd51df5864af9b8493acd69db0747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } }
I20250627 13:53:03.826974 5097 consensus_queue.cc:260] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:53:03.827409 5097 ts_tablet_manager.cc:1428] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250627 13:53:03.827505 5098 ts_tablet_manager.cc:1428] T a01527b9f7c14748837f5ad4dbaa9e9d P f46fd51df5864af9b8493acd69db0747: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250627 13:53:03.827736 5097 tablet_bootstrap.cc:492] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Bootstrap starting.
I20250627 13:53:03.827872 5096 tablet_bootstrap.cc:492] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab: Bootstrap starting.
I20250627 13:53:03.829407 5096 tablet_bootstrap.cc:654] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:03.829596 5097 tablet_bootstrap.cc:654] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:03.832999 5097 tablet_bootstrap.cc:492] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6: No bootstrap required, opened a new log
I20250627 13:53:03.833217 5097 ts_tablet_manager.cc:1397] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20250627 13:53:03.834532 5097 raft_consensus.cc:357] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:53:03.834769 5097 raft_consensus.cc:383] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:03.834854 5097 raft_consensus.cc:738] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb586b8cb7ab44e1a7ec5016dbd2c0a6, State: Initialized, Role: FOLLOWER
I20250627 13:53:03.835103 5097 consensus_queue.cc:260] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:53:03.835681 5097 ts_tablet_manager.cc:1428] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:53:03.837524 5096 tablet_bootstrap.cc:492] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab: No bootstrap required, opened a new log
I20250627 13:53:03.837718 5096 ts_tablet_manager.cc:1397] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20250627 13:53:03.838327 5096 raft_consensus.cc:357] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } }
I20250627 13:53:03.838517 5096 raft_consensus.cc:383] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:03.838603 5096 raft_consensus.cc:738] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad800aa8621b4fc5938692202990b5ab, State: Initialized, Role: FOLLOWER
I20250627 13:53:03.838848 5096 consensus_queue.cc:260] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } }
I20250627 13:53:03.839366 5096 ts_tablet_manager.cc:1428] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250627 13:53:03.839640 5096 tablet_bootstrap.cc:492] T 978e20547d644d3988d6da29235903aa P ad800aa8621b4fc5938692202990b5ab: Bootstrap starting.
I20250627 13:53:03.841297 5096 tablet_bootstrap.cc:654] T 978e20547d644d3988d6da29235903aa P ad800aa8621b4fc5938692202990b5ab: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:03.843046 5096 tablet_bootstrap.cc:492] T 978e20547d644d3988d6da29235903aa P ad800aa8621b4fc5938692202990b5ab: No bootstrap required, opened a new log
I20250627 13:53:03.843212 5096 ts_tablet_manager.cc:1397] T 978e20547d644d3988d6da29235903aa P ad800aa8621b4fc5938692202990b5ab: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250627 13:53:03.843868 5096 raft_consensus.cc:357] T 978e20547d644d3988d6da29235903aa P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:53:03.844024 5096 raft_consensus.cc:383] T 978e20547d644d3988d6da29235903aa P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:03.844096 5096 raft_consensus.cc:738] T 978e20547d644d3988d6da29235903aa P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad800aa8621b4fc5938692202990b5ab, State: Initialized, Role: FOLLOWER
I20250627 13:53:03.844252 5096 consensus_queue.cc:260] T 978e20547d644d3988d6da29235903aa P ad800aa8621b4fc5938692202990b5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:53:03.844580 5096 ts_tablet_manager.cc:1428] T 978e20547d644d3988d6da29235903aa P ad800aa8621b4fc5938692202990b5ab: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250627 13:53:03.922922 5078 raft_consensus.cc:491] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:03.923128 5078 raft_consensus.cc:513] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:53:03.923684 5078 leader_election.cc:290] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad800aa8621b4fc5938692202990b5ab (127.0.103.66:43273)
I20250627 13:53:03.924281 4063 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "978e20547d644d3988d6da29235903aa" candidate_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad800aa8621b4fc5938692202990b5ab" is_pre_election: true
I20250627 13:53:03.924459 4063 raft_consensus.cc:2466] T 978e20547d644d3988d6da29235903aa P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb586b8cb7ab44e1a7ec5016dbd2c0a6 in term 0.
I20250627 13:53:03.924837 4128 leader_election.cc:304] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ad800aa8621b4fc5938692202990b5ab, fb586b8cb7ab44e1a7ec5016dbd2c0a6; no voters:
I20250627 13:53:03.925084 5078 raft_consensus.cc:2802] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:03.925189 5078 raft_consensus.cc:491] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:03.925266 5078 raft_consensus.cc:3058] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:03.927248 5078 raft_consensus.cc:513] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:53:03.927699 5078 leader_election.cc:290] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 election: Requested vote from peers ad800aa8621b4fc5938692202990b5ab (127.0.103.66:43273)
I20250627 13:53:03.928272 4063 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "978e20547d644d3988d6da29235903aa" candidate_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad800aa8621b4fc5938692202990b5ab"
I20250627 13:53:03.928480 4063 raft_consensus.cc:3058] T 978e20547d644d3988d6da29235903aa P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:03.930578 4063 raft_consensus.cc:2466] T 978e20547d644d3988d6da29235903aa P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb586b8cb7ab44e1a7ec5016dbd2c0a6 in term 1.
I20250627 13:53:03.930965 4128 leader_election.cc:304] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ad800aa8621b4fc5938692202990b5ab, fb586b8cb7ab44e1a7ec5016dbd2c0a6; no voters:
I20250627 13:53:03.931272 5078 raft_consensus.cc:2802] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:03.931424 5078 raft_consensus.cc:695] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Becoming Leader. State: Replica: fb586b8cb7ab44e1a7ec5016dbd2c0a6, State: Running, Role: LEADER
I20250627 13:53:03.931692 5078 consensus_queue.cc:237] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:53:03.934077 3789 catalog_manager.cc:5582] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } health_report { overall_health: HEALTHY } } }
I20250627 13:53:04.001701 4965 raft_consensus.cc:491] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:04.001907 4965 raft_consensus.cc:513] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } }
I20250627 13:53:04.002514 4965 leader_election.cc:290] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133)
I20250627 13:53:04.003015 3932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a01527b9f7c14748837f5ad4dbaa9e9d" candidate_uuid: "ad800aa8621b4fc5938692202990b5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46fd51df5864af9b8493acd69db0747" is_pre_election: true
I20250627 13:53:04.003268 3932 raft_consensus.cc:2466] T a01527b9f7c14748837f5ad4dbaa9e9d P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad800aa8621b4fc5938692202990b5ab in term 0.
I20250627 13:53:04.003720 3997 leader_election.cc:304] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ad800aa8621b4fc5938692202990b5ab, f46fd51df5864af9b8493acd69db0747; no voters:
I20250627 13:53:04.003969 4965 raft_consensus.cc:2802] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:04.004107 4965 raft_consensus.cc:491] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:04.004222 4965 raft_consensus.cc:3058] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:04.006161 4965 raft_consensus.cc:513] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } }
I20250627 13:53:04.006570 4965 leader_election.cc:290] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab [CANDIDATE]: Term 1 election: Requested vote from peers f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133)
I20250627 13:53:04.007078 3932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a01527b9f7c14748837f5ad4dbaa9e9d" candidate_uuid: "ad800aa8621b4fc5938692202990b5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f46fd51df5864af9b8493acd69db0747"
I20250627 13:53:04.007251 3932 raft_consensus.cc:3058] T a01527b9f7c14748837f5ad4dbaa9e9d P f46fd51df5864af9b8493acd69db0747 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:04.009101 3932 raft_consensus.cc:2466] T a01527b9f7c14748837f5ad4dbaa9e9d P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad800aa8621b4fc5938692202990b5ab in term 1.
I20250627 13:53:04.009491 3997 leader_election.cc:304] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ad800aa8621b4fc5938692202990b5ab, f46fd51df5864af9b8493acd69db0747; no voters:
I20250627 13:53:04.009760 4965 raft_consensus.cc:2802] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:04.010105 4965 raft_consensus.cc:695] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab [term 1 LEADER]: Becoming Leader. State: Replica: ad800aa8621b4fc5938692202990b5ab, State: Running, Role: LEADER
I20250627 13:53:04.010351 4965 consensus_queue.cc:237] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } }
I20250627 13:53:04.012871 3789 catalog_manager.cc:5582] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab reported cstate change: term changed from 0 to 1, leader changed from <none> to ad800aa8621b4fc5938692202990b5ab (127.0.103.66). New cstate: current_term: 1 leader_uuid: "ad800aa8621b4fc5938692202990b5ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } health_report { overall_health: HEALTHY } } }
I20250627 13:53:04.143179 5078 raft_consensus.cc:491] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:04.143428 5078 raft_consensus.cc:513] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:53:04.143955 5078 leader_election.cc:290] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad800aa8621b4fc5938692202990b5ab (127.0.103.66:43273)
I20250627 13:53:04.144615 4063 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8fe13ad746124b1d82cfb8ce691828a4" candidate_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad800aa8621b4fc5938692202990b5ab" is_pre_election: true
I20250627 13:53:04.144820 4063 raft_consensus.cc:2466] T 8fe13ad746124b1d82cfb8ce691828a4 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb586b8cb7ab44e1a7ec5016dbd2c0a6 in term 0.
I20250627 13:53:04.145253 4128 leader_election.cc:304] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ad800aa8621b4fc5938692202990b5ab, fb586b8cb7ab44e1a7ec5016dbd2c0a6; no voters:
I20250627 13:53:04.145552 5078 raft_consensus.cc:2802] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:04.145666 5078 raft_consensus.cc:491] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:04.145788 5078 raft_consensus.cc:3058] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:04.147699 5078 raft_consensus.cc:513] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:53:04.148093 5078 leader_election.cc:290] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 election: Requested vote from peers ad800aa8621b4fc5938692202990b5ab (127.0.103.66:43273)
I20250627 13:53:04.148625 4063 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8fe13ad746124b1d82cfb8ce691828a4" candidate_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad800aa8621b4fc5938692202990b5ab"
I20250627 13:53:04.148813 4063 raft_consensus.cc:3058] T 8fe13ad746124b1d82cfb8ce691828a4 P ad800aa8621b4fc5938692202990b5ab [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:04.150889 4063 raft_consensus.cc:2466] T 8fe13ad746124b1d82cfb8ce691828a4 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb586b8cb7ab44e1a7ec5016dbd2c0a6 in term 1.
I20250627 13:53:04.151297 4128 leader_election.cc:304] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ad800aa8621b4fc5938692202990b5ab, fb586b8cb7ab44e1a7ec5016dbd2c0a6; no voters:
I20250627 13:53:04.151578 5078 raft_consensus.cc:2802] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:04.151731 5078 raft_consensus.cc:695] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Becoming Leader. State: Replica: fb586b8cb7ab44e1a7ec5016dbd2c0a6, State: Running, Role: LEADER
I20250627 13:53:04.151968 5078 consensus_queue.cc:237] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } }
I20250627 13:53:04.154317 3789 catalog_manager.cc:5582] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } health_report { overall_health: HEALTHY } } }
I20250627 13:53:04.469192 5078 consensus_queue.cc:1035] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:04.497020 4965 consensus_queue.cc:1035] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250627 13:53:04.502652 5109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:53:04.503027 5109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:53:04.510996 5109 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20250627 13:53:04.658492 4063 raft_consensus.cc:1273] T 8fe13ad746124b1d82cfb8ce691828a4 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250627 13:53:04.659741 5078 consensus_queue.cc:1035] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 182.543 ms
time per row: 0.0912717 ms
I20250627 13:53:04.839119 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.68:45503
--local_ip_for_outbound_sockets=127.0.103.68
--tserver_master_addrs=127.0.103.126:39869
--webserver_port=46313
--webserver_interface=127.0.103.68
--builtin_ntp_servers=127.0.103.84:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250627 13:53:05.169335 5159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:53:05.169637 5159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:53:05.169801 5159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:53:05.177359 5159 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:53:05.177459 5159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:53:05.177583 5159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.68
I20250627 13:53:05.187906 5159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.68:45503
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.103.68
--webserver_port=46313
--tserver_master_addrs=127.0.103.126:39869
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.68
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:53:05.189884 5159 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:53:05.192122 5159 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:53:05.205549 5167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:53:05.205482 5165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:53:05.205633 5164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:53:05.206012 5159 server_base.cc:1048] running on GCE node
I20250627 13:53:05.206802 5159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:53:05.207688 5159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:53:05.208891 5159 hybrid_clock.cc:648] HybridClock initialized: now 1751032385208836 us; error 38 us; skew 500 ppm
I20250627 13:53:05.209157 5159 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:53:05.211252 5159 webserver.cc:469] Webserver started at http://127.0.103.68:46313/ using document root <none> and password file <none>
I20250627 13:53:05.211927 5159 fs_manager.cc:362] Metadata directory not provided
I20250627 13:53:05.212059 5159 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:53:05.217159 5159 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20250627 13:53:05.219882 5173 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:05.221104 5159 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20250627 13:53:05.221249 5159 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "36100389495d4971b10cf590bc82c42c"
format_stamp: "Formatted at 2025-06-27 13:52:51 on dist-test-slave-1ckg"
I20250627 13:53:05.221985 5159 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:53:05.246742 5159 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:53:05.247591 5159 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:53:05.247833 5159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:53:05.248723 5159 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:53:05.250349 5159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:53:05.250448 5159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:05.250560 5159 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:53:05.250630 5159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:05.283761 5159 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:45503
I20250627 13:53:05.283830 5285 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:45503 every 8 connection(s)
I20250627 13:53:05.285307 5159 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250627 13:53:05.287185 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 5159
I20250627 13:53:05.287869 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.69:42665
--local_ip_for_outbound_sockets=127.0.103.69
--tserver_master_addrs=127.0.103.126:39869
--webserver_port=41177
--webserver_interface=127.0.103.69
--builtin_ntp_servers=127.0.103.84:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:53:05.300251 5286 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39869
I20250627 13:53:05.300601 5286 heartbeater.cc:461] Registering TS with master...
I20250627 13:53:05.301550 5286 heartbeater.cc:507] Master 127.0.103.126:39869 requested a full tablet report, sending...
I20250627 13:53:05.303515 3788 ts_manager.cc:194] Re-registered known tserver with Master: 36100389495d4971b10cf590bc82c42c (127.0.103.68:45503)
I20250627 13:53:05.304847 3788 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.68:54507
W20250627 13:53:05.619927 5290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:53:05.620267 5290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:53:05.620352 5290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:53:05.628422 5290 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:53:05.628530 5290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:53:05.628686 5290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.69
I20250627 13:53:05.638149 5290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.69:42665
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.103.69
--webserver_port=41177
--tserver_master_addrs=127.0.103.126:39869
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.69
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:53:05.639701 5290 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:53:05.641462 5290 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:53:05.652945 5296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:53:05.652984 5298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:53:05.653033 5295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:53:05.653921 5290 server_base.cc:1048] running on GCE node
I20250627 13:53:05.654511 5290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:53:05.655336 5290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:53:05.656560 5290 hybrid_clock.cc:648] HybridClock initialized: now 1751032385656496 us; error 41 us; skew 500 ppm
I20250627 13:53:05.656814 5290 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:53:05.663448 5290 webserver.cc:469] Webserver started at http://127.0.103.69:41177/ using document root <none> and password file <none>
I20250627 13:53:05.664197 5290 fs_manager.cc:362] Metadata directory not provided
I20250627 13:53:05.664341 5290 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:53:05.669890 5290 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20250627 13:53:05.672437 5304 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:05.673731 5290 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250627 13:53:05.673856 5290 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "911714a383074e09a9b2d127ab46d2c1"
format_stamp: "Formatted at 2025-06-27 13:52:52 on dist-test-slave-1ckg"
I20250627 13:53:05.674458 5290 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:53:05.694963 5290 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:53:05.695987 5290 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:53:05.696195 5290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:53:05.697108 5290 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:53:05.698606 5290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:53:05.698691 5290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:05.698765 5290 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:53:05.698815 5290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:05.732204 5290 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.69:42665
I20250627 13:53:05.732262 5416 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.69:42665 every 8 connection(s)
I20250627 13:53:05.733628 5290 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250627 13:53:05.737527 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 5290
I20250627 13:53:05.738199 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.70:37215
--local_ip_for_outbound_sockets=127.0.103.70
--tserver_master_addrs=127.0.103.126:39869
--webserver_port=43963
--webserver_interface=127.0.103.70
--builtin_ntp_servers=127.0.103.84:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:53:05.747821 5417 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39869
I20250627 13:53:05.748167 5417 heartbeater.cc:461] Registering TS with master...
I20250627 13:53:05.748986 5417 heartbeater.cc:507] Master 127.0.103.126:39869 requested a full tablet report, sending...
I20250627 13:53:05.750409 3788 ts_manager.cc:194] Re-registered known tserver with Master: 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69:42665)
I20250627 13:53:05.751178 3788 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.69:52581
W20250627 13:53:06.050861 5420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:53:06.051182 5420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:53:06.051313 5420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:53:06.059235 5420 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:53:06.059373 5420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:53:06.059702 5420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.70
I20250627 13:53:06.069545 5420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.70:37215
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.103.70
--webserver_port=43963
--tserver_master_addrs=127.0.103.126:39869
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.70
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:53:06.071259 5420 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:53:06.073030 5420 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:53:06.084641 5428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:53:06.084750 5425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:53:06.084615 5426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:53:06.085601 5420 server_base.cc:1048] running on GCE node
I20250627 13:53:06.086210 5420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:53:06.087138 5420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:53:06.088364 5420 hybrid_clock.cc:648] HybridClock initialized: now 1751032386088293 us; error 49 us; skew 500 ppm
I20250627 13:53:06.088640 5420 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:53:06.090898 5420 webserver.cc:469] Webserver started at http://127.0.103.70:43963/ using document root <none> and password file <none>
I20250627 13:53:06.091629 5420 fs_manager.cc:362] Metadata directory not provided
I20250627 13:53:06.091768 5420 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:53:06.097492 5420 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250627 13:53:06.100256 5434 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:06.101755 5420 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250627 13:53:06.101963 5420 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "bf0be937c3634bcab62f1e58402a776f"
format_stamp: "Formatted at 2025-06-27 13:52:52 on dist-test-slave-1ckg"
I20250627 13:53:06.102732 5420 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:53:06.118315 5420 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:53:06.119166 5420 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:53:06.119437 5420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:53:06.120411 5420 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:53:06.122040 5420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:53:06.122140 5420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:06.122263 5420 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:53:06.122349 5420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:06.160604 5420 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.70:37215
I20250627 13:53:06.160660 5546 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.70:37215 every 8 connection(s)
I20250627 13:53:06.162535 5420 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20250627 13:53:06.167445 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 5420
I20250627 13:53:06.167932 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.71:45413
--local_ip_for_outbound_sockets=127.0.103.71
--tserver_master_addrs=127.0.103.126:39869
--webserver_port=34299
--webserver_interface=127.0.103.71
--builtin_ntp_servers=127.0.103.84:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:53:06.175575 5547 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39869
I20250627 13:53:06.175909 5547 heartbeater.cc:461] Registering TS with master...
I20250627 13:53:06.176692 5547 heartbeater.cc:507] Master 127.0.103.126:39869 requested a full tablet report, sending...
I20250627 13:53:06.178151 3788 ts_manager.cc:194] Re-registered known tserver with Master: bf0be937c3634bcab62f1e58402a776f (127.0.103.70:37215)
I20250627 13:53:06.179008 3788 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.70:52881
I20250627 13:53:06.307696 5286 heartbeater.cc:499] Master 127.0.103.126:39869 was elected leader, sending a full tablet report...
W20250627 13:53:06.490746 5550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:53:06.491066 5550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:53:06.491191 5550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:53:06.499215 5550 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:53:06.499349 5550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:53:06.499552 5550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.71
I20250627 13:53:06.510097 5550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.71:45413
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.103.71
--webserver_port=34299
--tserver_master_addrs=127.0.103.126:39869
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.71
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:53:06.511952 5550 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:53:06.513731 5550 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:53:06.525326 5556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:53:06.525259 5558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:53:06.525305 5555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:53:06.526327 5550 server_base.cc:1048] running on GCE node
I20250627 13:53:06.526893 5550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:53:06.527809 5550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:53:06.529093 5550 hybrid_clock.cc:648] HybridClock initialized: now 1751032386529013 us; error 51 us; skew 500 ppm
I20250627 13:53:06.529426 5550 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:53:06.531836 5550 webserver.cc:469] Webserver started at http://127.0.103.71:34299/ using document root <none> and password file <none>
I20250627 13:53:06.532577 5550 fs_manager.cc:362] Metadata directory not provided
I20250627 13:53:06.532711 5550 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:53:06.538396 5550 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20250627 13:53:06.541244 5564 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:06.542634 5550 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250627 13:53:06.542793 5550 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "6503b59977444aaebe5ee70a0bc99154"
format_stamp: "Formatted at 2025-06-27 13:52:53 on dist-test-slave-1ckg"
I20250627 13:53:06.543498 5550 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:53:06.565371 5550 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:53:06.566347 5550 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:53:06.566618 5550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:53:06.567762 5550 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:53:06.569917 5571 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250627 13:53:06.577208 5550 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250627 13:53:06.577338 5550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.000s
I20250627 13:53:06.577473 5550 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250627 13:53:06.580619 5550 ts_tablet_manager.cc:610] Registered 1 tablets
I20250627 13:53:06.580715 5550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20250627 13:53:06.581116 5571 tablet_bootstrap.cc:492] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154: Bootstrap starting.
I20250627 13:53:06.600502 5571 log.cc:826] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154: Log is configured to *not* fsync() on all Append() calls
I20250627 13:53:06.605139 5571 tablet_bootstrap.cc:492] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250627 13:53:06.606328 5571 tablet_bootstrap.cc:492] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154: Bootstrap complete.
I20250627 13:53:06.607560 5571 ts_tablet_manager.cc:1397] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154: Time spent bootstrapping tablet: real 0.027s user 0.020s sys 0.004s
I20250627 13:53:06.615976 5571 raft_consensus.cc:357] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:06.617372 5571 raft_consensus.cc:738] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6503b59977444aaebe5ee70a0bc99154, State: Initialized, Role: FOLLOWER
I20250627 13:53:06.618327 5571 consensus_queue.cc:260] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:06.620010 5571 ts_tablet_manager.cc:1428] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20250627 13:53:06.632464 5550 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.71:45413
I20250627 13:53:06.632505 5679 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.71:45413 every 8 connection(s)
I20250627 13:53:06.633908 5550 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data/info.pb
W20250627 13:53:06.635130 5681 tablet.cc:2378] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:53:06.637367 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 5550
I20250627 13:53:06.637714 413 test_util.cc:276] Using random seed: -10137726
I20250627 13:53:06.646262 5680 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39869
I20250627 13:53:06.646601 5680 heartbeater.cc:461] Registering TS with master...
I20250627 13:53:06.647447 5680 heartbeater.cc:507] Master 127.0.103.126:39869 requested a full tablet report, sending...
I20250627 13:53:06.650064 3788 ts_manager.cc:194] Re-registered known tserver with Master: 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71:45413)
I20250627 13:53:06.651544 3788 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.71:38107
I20250627 13:53:06.656260 413 test_workload.cc:405] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20250627 13:53:06.656704 413 test_util.cc:276] Using random seed: -10118744
I20250627 13:53:06.671245 413 test_workload.cc:405] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20250627 13:53:06.671624 413 test_util.cc:276] Using random seed: -10103854
I20250627 13:53:06.689253 413 test_workload.cc:405] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20250627 13:53:06.689689 413 test_util.cc:276] Using random seed: -10085791
I20250627 13:53:06.753576 5417 heartbeater.cc:499] Master 127.0.103.126:39869 was elected leader, sending a full tablet report...
I20250627 13:53:06.789604 413 test_workload.cc:405] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20250627 13:53:06.806890 413 test_util.cc:276] Using random seed: -9968583
I20250627 13:53:06.879057 413 test_workload.cc:405] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20250627 13:53:07.181557 5547 heartbeater.cc:499] Master 127.0.103.126:39869 was elected leader, sending a full tablet report...
I20250627 13:53:07.654565 5680 heartbeater.cc:499] Master 127.0.103.126:39869 was elected leader, sending a full tablet report...
I20250627 13:53:08.365906 5221 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:53:08.438172 5829 raft_consensus.cc:491] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:08.438654 5829 raft_consensus.cc:513] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:08.447427 5829 leader_election.cc:290] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133)
I20250627 13:53:08.463940 3930 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fdcc98da13c4cf998b1fd1b3e3f9fe5" candidate_uuid: "6503b59977444aaebe5ee70a0bc99154" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f46fd51df5864af9b8493acd69db0747" is_pre_election: true
I20250627 13:53:08.464231 3930 raft_consensus.cc:2466] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6503b59977444aaebe5ee70a0bc99154 in term 1.
I20250627 13:53:08.464883 5565 leader_election.cc:304] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6503b59977444aaebe5ee70a0bc99154, f46fd51df5864af9b8493acd69db0747; no voters:
I20250627 13:53:08.466646 5829 raft_consensus.cc:2802] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250627 13:53:08.466912 5829 raft_consensus.cc:491] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:08.467087 5829 raft_consensus.cc:3058] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:53:08.472142 5829 raft_consensus.cc:513] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:08.474283 5829 leader_election.cc:290] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [CANDIDATE]: Term 2 election: Requested vote from peers f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133)
I20250627 13:53:08.474545 3930 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fdcc98da13c4cf998b1fd1b3e3f9fe5" candidate_uuid: "6503b59977444aaebe5ee70a0bc99154" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f46fd51df5864af9b8493acd69db0747"
I20250627 13:53:08.474735 3930 raft_consensus.cc:3058] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:53:08.477695 3930 raft_consensus.cc:2466] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6503b59977444aaebe5ee70a0bc99154 in term 2.
I20250627 13:53:08.478106 5565 leader_election.cc:304] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6503b59977444aaebe5ee70a0bc99154, f46fd51df5864af9b8493acd69db0747; no voters:
I20250627 13:53:08.479494 5829 raft_consensus.cc:2802] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:53:08.481140 5482 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:53:08.490515 5829 raft_consensus.cc:695] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [term 2 LEADER]: Becoming Leader. State: Replica: 6503b59977444aaebe5ee70a0bc99154, State: Running, Role: LEADER
I20250627 13:53:08.491056 5829 consensus_queue.cc:237] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } }
I20250627 13:53:08.494892 4174 tablet_service.cc:1430] Tablet server has 9 leaders and 0 scanners
I20250627 13:53:08.502940 3789 catalog_manager.cc:5582] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "6503b59977444aaebe5ee70a0bc99154" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:08.521842 5614 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:08.523715 5352 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:53:08.534562 3912 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
I20250627 13:53:08.672892 4042 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:53:08.891804 3932 raft_consensus.cc:1273] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [term 2 FOLLOWER]: Refusing update from remote peer 6503b59977444aaebe5ee70a0bc99154: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250627 13:53:08.892760 5829 consensus_queue.cc:1035] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:09.550714 5221 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:53:09.632320 5614 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:09.633744 5352 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:53:09.686398 4042 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:53:09.725571 5482 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:53:09.765040 4174 tablet_service.cc:1430] Tablet server has 9 leaders and 0 scanners
I20250627 13:53:09.819504 3912 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
I20250627 13:53:10.038764 3931 consensus_queue.cc:237] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 1.61, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: true } }
I20250627 13:53:10.041443 4062 raft_consensus.cc:1273] T 40b04be88e9947c5a5ca8432e3791bdd P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Refusing update from remote peer f46fd51df5864af9b8493acd69db0747: Log matching property violated. Preceding OpId in replica: term: 1 index: 61. Preceding OpId from leader: term: 1 index: 62. (index mismatch)
W20250627 13:53:10.057940 3869 consensus_peers.cc:487] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 -> Peer bf0be937c3634bcab62f1e58402a776f (127.0.103.70:37215): Couldn't send request to peer bf0be937c3634bcab62f1e58402a776f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40b04be88e9947c5a5ca8432e3791bdd. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:53:10.059422 5011 consensus_queue.cc:1035] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 60, Time since last communication: 0.000s
I20250627 13:53:10.070981 5882 raft_consensus.cc:2953] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 [term 1 LEADER]: Committing config change with OpId 1.62: config changed from index -1 to 62, NON_VOTER bf0be937c3634bcab62f1e58402a776f (127.0.103.70) added. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: true } } }
I20250627 13:53:10.075502 4060 raft_consensus.cc:2953] T 40b04be88e9947c5a5ca8432e3791bdd P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Committing config change with OpId 1.62: config changed from index -1 to 62, NON_VOTER bf0be937c3634bcab62f1e58402a776f (127.0.103.70) added. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: true } } }
I20250627 13:53:10.076735 3788 catalog_manager.cc:5582] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 reported cstate change: config changed from index -1 to 62, NON_VOTER bf0be937c3634bcab62f1e58402a776f (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "f46fd51df5864af9b8493acd69db0747" committed_config { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:10.247574 4193 consensus_queue.cc:237] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 1.65, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: true } }
I20250627 13:53:10.254283 3930 raft_consensus.cc:1273] T 1cd6f75e462546c6bb05ef14af3498fa P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20250627 13:53:10.257609 5710 consensus_queue.cc:1035] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
W20250627 13:53:10.261418 4128 consensus_peers.cc:487] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 -> Peer 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69:42665): Couldn't send request to peer 911714a383074e09a9b2d127ab46d2c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1cd6f75e462546c6bb05ef14af3498fa. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:53:10.265270 5850 raft_consensus.cc:2953] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.66: config changed from index -1 to 66, NON_VOTER 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69) added. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: true } } }
I20250627 13:53:10.271387 3930 raft_consensus.cc:2953] T 1cd6f75e462546c6bb05ef14af3498fa P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index -1 to 66, NON_VOTER 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69) added. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: true } } }
I20250627 13:53:10.277038 3789 catalog_manager.cc:5582] T 1cd6f75e462546c6bb05ef14af3498fa P f46fd51df5864af9b8493acd69db0747 reported cstate change: config changed from index -1 to 66, NON_VOTER 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69) added. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: true } } }
I20250627 13:53:10.354578 4193 consensus_queue.cc:237] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 1.78, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: true } }
I20250627 13:53:10.362068 4060 raft_consensus.cc:1273] T 8fe13ad746124b1d82cfb8ce691828a4 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20250627 13:53:10.363130 5850 consensus_queue.cc:1035] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.001s
I20250627 13:53:10.372689 5078 raft_consensus.cc:2953] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.79: config changed from index -1 to 79, NON_VOTER 36100389495d4971b10cf590bc82c42c (127.0.103.68) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: true } } }
I20250627 13:53:10.375526 4060 raft_consensus.cc:2953] T 8fe13ad746124b1d82cfb8ce691828a4 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index -1 to 79, NON_VOTER 36100389495d4971b10cf590bc82c42c (127.0.103.68) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: true } } }
I20250627 13:53:10.380735 3788 catalog_manager.cc:5582] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 reported cstate change: config changed from index -1 to 79, NON_VOTER 36100389495d4971b10cf590bc82c42c (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250627 13:53:10.394626 4129 consensus_peers.cc:487] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 -> Peer 36100389495d4971b10cf590bc82c42c (127.0.103.68:45503): Couldn't send request to peer 36100389495d4971b10cf590bc82c42c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8fe13ad746124b1d82cfb8ce691828a4. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:53:10.410012 4193 consensus_queue.cc:237] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 1.73, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: true } }
I20250627 13:53:10.411756 3933 raft_consensus.cc:1273] T be42f2b9b1b747adb641149f70da608a P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20250627 13:53:10.412655 5850 consensus_queue.cc:1035] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20250627 13:53:10.415746 5767 raft_consensus.cc:2953] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.74: config changed from index -1 to 74, NON_VOTER 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) added. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: true } } }
I20250627 13:53:10.420861 3933 raft_consensus.cc:2953] T be42f2b9b1b747adb641149f70da608a P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index -1 to 74, NON_VOTER 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) added. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: true } } }
I20250627 13:53:10.424428 3788 catalog_manager.cc:5582] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 reported cstate change: config changed from index -1 to 74, NON_VOTER 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) added. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250627 13:53:10.481068 4128 consensus_peers.cc:487] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 -> Peer 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71:45413): Couldn't send request to peer 6503b59977444aaebe5ee70a0bc99154. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: be42f2b9b1b747adb641149f70da608a. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:53:10.488718 4194 consensus_queue.cc:237] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 1.88, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: true } }
I20250627 13:53:10.493171 3933 raft_consensus.cc:1273] T 3e2b11c6788e4ccdbdde60769fee1aae P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 88. Preceding OpId from leader: term: 1 index: 89. (index mismatch)
I20250627 13:53:10.493878 5850 consensus_queue.cc:1035] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 87, Time since last communication: 0.000s
W20250627 13:53:10.497524 4130 consensus_peers.cc:487] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 -> Peer bf0be937c3634bcab62f1e58402a776f (127.0.103.70:37215): Couldn't send request to peer bf0be937c3634bcab62f1e58402a776f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e2b11c6788e4ccdbdde60769fee1aae. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:53:10.498045 5078 raft_consensus.cc:2953] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.89: config changed from index -1 to 89, NON_VOTER bf0be937c3634bcab62f1e58402a776f (127.0.103.70) added. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: true } } }
I20250627 13:53:10.508864 3930 raft_consensus.cc:2953] T 3e2b11c6788e4ccdbdde60769fee1aae P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Committing config change with OpId 1.89: config changed from index -1 to 89, NON_VOTER bf0be937c3634bcab62f1e58402a776f (127.0.103.70) added. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: true } } }
I20250627 13:53:10.514217 3788 catalog_manager.cc:5582] T 3e2b11c6788e4ccdbdde60769fee1aae P f46fd51df5864af9b8493acd69db0747 reported cstate change: config changed from index -1 to 89, NON_VOTER bf0be937c3634bcab62f1e58402a776f (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: true } } }
I20250627 13:53:10.635128 4062 consensus_queue.cc:237] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 1.91, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: true } }
I20250627 13:53:10.663125 4194 raft_consensus.cc:1273] T 37f6cffc2bc5456e94999b60340dfeb6 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Refusing update from remote peer ad800aa8621b4fc5938692202990b5ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 91. Preceding OpId from leader: term: 1 index: 92. (index mismatch)
W20250627 13:53:10.671121 3998 consensus_peers.cc:487] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab -> Peer 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69:42665): Couldn't send request to peer 911714a383074e09a9b2d127ab46d2c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37f6cffc2bc5456e94999b60340dfeb6. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:53:10.673339 4965 consensus_queue.cc:1035] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20250627 13:53:10.685545 4965 raft_consensus.cc:2953] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [term 1 LEADER]: Committing config change with OpId 1.92: config changed from index -1 to 92, NON_VOTER 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69) added. New config: { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: true } } }
I20250627 13:53:10.688676 4194 raft_consensus.cc:2953] T 37f6cffc2bc5456e94999b60340dfeb6 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Committing config change with OpId 1.92: config changed from index -1 to 92, NON_VOTER 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69) added. New config: { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: true } } }
I20250627 13:53:10.697863 3788 catalog_manager.cc:5582] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab reported cstate change: config changed from index -1 to 92, NON_VOTER 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69) added. New cstate: current_term: 1 leader_uuid: "ad800aa8621b4fc5938692202990b5ab" committed_config { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:10.789494 5957 ts_tablet_manager.cc:927] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c: Initiating tablet copy from peer fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67:44627)
I20250627 13:53:10.791946 5952 ts_tablet_manager.cc:927] T 40b04be88e9947c5a5ca8432e3791bdd P bf0be937c3634bcab62f1e58402a776f: Initiating tablet copy from peer f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133)
I20250627 13:53:10.792642 5953 ts_tablet_manager.cc:927] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1: Initiating tablet copy from peer fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67:44627)
I20250627 13:53:10.798027 5957 tablet_copy_client.cc:323] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:44627
I20250627 13:53:10.797977 4194 consensus_queue.cc:237] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 1.70, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: true } }
W20250627 13:53:10.800508 4129 consensus_peers.cc:487] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 -> Peer 36100389495d4971b10cf590bc82c42c (127.0.103.68:45503): Couldn't send request to peer 36100389495d4971b10cf590bc82c42c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c20692dbde954a259acdb9c765978113. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:53:10.807338 5952 tablet_copy_client.cc:323] T 40b04be88e9947c5a5ca8432e3791bdd P bf0be937c3634bcab62f1e58402a776f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:35133
I20250627 13:53:10.812067 4062 raft_consensus.cc:1273] T c20692dbde954a259acdb9c765978113 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 71. (index mismatch)
I20250627 13:53:10.818770 5940 consensus_queue.cc:1035] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20250627 13:53:10.829072 5850 raft_consensus.cc:2953] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.71: config changed from index -1 to 71, NON_VOTER 36100389495d4971b10cf590bc82c42c (127.0.103.68) added. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: true } } }
I20250627 13:53:10.836325 4064 raft_consensus.cc:2953] T c20692dbde954a259acdb9c765978113 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index -1 to 71, NON_VOTER 36100389495d4971b10cf590bc82c42c (127.0.103.68) added. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: true } } }
I20250627 13:53:10.836462 3789 catalog_manager.cc:5582] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 reported cstate change: config changed from index -1 to 71, NON_VOTER 36100389495d4971b10cf590bc82c42c (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:10.854849 4214 tablet_copy_service.cc:140] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Received BeginTabletCopySession request for tablet 8fe13ad746124b1d82cfb8ce691828a4 from peer 36100389495d4971b10cf590bc82c42c ({username='slave'} at 127.0.103.68:36407)
I20250627 13:53:10.855186 4214 tablet_copy_service.cc:161] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Beginning new tablet copy session on tablet 8fe13ad746124b1d82cfb8ce691828a4 from peer 36100389495d4971b10cf590bc82c42c at {username='slave'} at 127.0.103.68:36407: session id = 36100389495d4971b10cf590bc82c42c-8fe13ad746124b1d82cfb8ce691828a4
I20250627 13:53:10.860337 4214 tablet_copy_source_session.cc:215] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:53:10.866331 5953 tablet_copy_client.cc:323] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:44627
I20250627 13:53:10.867749 5957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fe13ad746124b1d82cfb8ce691828a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:10.884856 5967 ts_tablet_manager.cc:927] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154: Initiating tablet copy from peer fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67:44627)
I20250627 13:53:10.888633 4214 tablet_copy_service.cc:140] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Received BeginTabletCopySession request for tablet 1cd6f75e462546c6bb05ef14af3498fa from peer 911714a383074e09a9b2d127ab46d2c1 ({username='slave'} at 127.0.103.69:36137)
I20250627 13:53:10.888798 4214 tablet_copy_service.cc:161] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Beginning new tablet copy session on tablet 1cd6f75e462546c6bb05ef14af3498fa from peer 911714a383074e09a9b2d127ab46d2c1 at {username='slave'} at 127.0.103.69:36137: session id = 911714a383074e09a9b2d127ab46d2c1-1cd6f75e462546c6bb05ef14af3498fa
I20250627 13:53:10.891558 4214 tablet_copy_source_session.cc:215] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:53:10.895871 5957 tablet_copy_client.cc:806] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c: tablet copy: Starting download of 0 data blocks...
I20250627 13:53:10.896245 5957 tablet_copy_client.cc:670] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c: tablet copy: Starting download of 1 WAL segments...
I20250627 13:53:10.899124 5957 tablet_copy_client.cc:538] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:53:10.903899 5957 tablet_bootstrap.cc:492] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c: Bootstrap starting.
I20250627 13:53:10.922047 5967 tablet_copy_client.cc:323] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:44627
I20250627 13:53:10.936365 3953 tablet_copy_service.cc:140] P f46fd51df5864af9b8493acd69db0747: Received BeginTabletCopySession request for tablet 40b04be88e9947c5a5ca8432e3791bdd from peer bf0be937c3634bcab62f1e58402a776f ({username='slave'} at 127.0.103.70:58947)
I20250627 13:53:10.936661 3953 tablet_copy_service.cc:161] P f46fd51df5864af9b8493acd69db0747: Beginning new tablet copy session on tablet 40b04be88e9947c5a5ca8432e3791bdd from peer bf0be937c3634bcab62f1e58402a776f at {username='slave'} at 127.0.103.70:58947: session id = bf0be937c3634bcab62f1e58402a776f-40b04be88e9947c5a5ca8432e3791bdd
I20250627 13:53:10.937357 5957 log.cc:826] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c: Log is configured to *not* fsync() on all Append() calls
I20250627 13:53:10.945067 3953 tablet_copy_source_session.cc:215] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:53:10.950066 5972 ts_tablet_manager.cc:927] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f: Initiating tablet copy from peer fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67:44627)
I20250627 13:53:10.978480 4214 tablet_copy_service.cc:140] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Received BeginTabletCopySession request for tablet be42f2b9b1b747adb641149f70da608a from peer 6503b59977444aaebe5ee70a0bc99154 ({username='slave'} at 127.0.103.71:57017)
I20250627 13:53:10.978652 4214 tablet_copy_service.cc:161] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Beginning new tablet copy session on tablet be42f2b9b1b747adb641149f70da608a from peer 6503b59977444aaebe5ee70a0bc99154 at {username='slave'} at 127.0.103.71:57017: session id = 6503b59977444aaebe5ee70a0bc99154-be42f2b9b1b747adb641149f70da608a
I20250627 13:53:10.981586 4214 tablet_copy_source_session.cc:215] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:53:10.984097 5967 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be42f2b9b1b747adb641149f70da608a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:10.984117 5952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40b04be88e9947c5a5ca8432e3791bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:10.992425 5967 tablet_copy_client.cc:806] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154: tablet copy: Starting download of 0 data blocks...
I20250627 13:53:10.992775 5953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cd6f75e462546c6bb05ef14af3498fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:10.992794 5967 tablet_copy_client.cc:670] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154: tablet copy: Starting download of 1 WAL segments...
I20250627 13:53:11.001528 5967 tablet_copy_client.cc:538] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:53:11.005563 5967 tablet_bootstrap.cc:492] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154: Bootstrap starting.
I20250627 13:53:11.025880 5972 tablet_copy_client.cc:323] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:44627
I20250627 13:53:11.034263 4214 tablet_copy_service.cc:140] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Received BeginTabletCopySession request for tablet 3e2b11c6788e4ccdbdde60769fee1aae from peer bf0be937c3634bcab62f1e58402a776f ({username='slave'} at 127.0.103.70:49887)
I20250627 13:53:11.034418 4214 tablet_copy_service.cc:161] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Beginning new tablet copy session on tablet 3e2b11c6788e4ccdbdde60769fee1aae from peer bf0be937c3634bcab62f1e58402a776f at {username='slave'} at 127.0.103.70:49887: session id = bf0be937c3634bcab62f1e58402a776f-3e2b11c6788e4ccdbdde60769fee1aae
I20250627 13:53:11.037570 4214 tablet_copy_source_session.cc:215] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:53:11.037739 4064 consensus_queue.cc:237] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 1.86, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: true } }
I20250627 13:53:11.040252 5972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e2b11c6788e4ccdbdde60769fee1aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:11.047631 5972 tablet_copy_client.cc:806] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f: tablet copy: Starting download of 0 data blocks...
I20250627 13:53:11.048027 5972 tablet_copy_client.cc:670] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f: tablet copy: Starting download of 1 WAL segments...
I20250627 13:53:11.050752 5972 tablet_copy_client.cc:538] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:53:11.055028 5972 tablet_bootstrap.cc:492] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f: Bootstrap starting.
W20250627 13:53:11.058557 3867 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.66:43273, user_credentials={real_user=slave}} blocked reactor thread for 61856.6us
I20250627 13:53:11.058848 5979 ts_tablet_manager.cc:927] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1: Initiating tablet copy from peer ad800aa8621b4fc5938692202990b5ab (127.0.103.66:43273)
I20250627 13:53:11.066176 3933 raft_consensus.cc:1273] T 76518bc59dfe443abd8e97539d1bcb8a P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Refusing update from remote peer ad800aa8621b4fc5938692202990b5ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 87. (index mismatch)
I20250627 13:53:11.059326 5952 tablet_copy_client.cc:806] T 40b04be88e9947c5a5ca8432e3791bdd P bf0be937c3634bcab62f1e58402a776f: tablet copy: Starting download of 0 data blocks...
I20250627 13:53:11.077164 5952 tablet_copy_client.cc:670] T 40b04be88e9947c5a5ca8432e3791bdd P bf0be937c3634bcab62f1e58402a776f: tablet copy: Starting download of 1 WAL segments...
I20250627 13:53:11.078667 5979 tablet_copy_client.cc:323] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:43273
W20250627 13:53:11.084005 3998 consensus_peers.cc:487] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab -> Peer 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71:45413): Couldn't send request to peer 6503b59977444aaebe5ee70a0bc99154. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 76518bc59dfe443abd8e97539d1bcb8a. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:53:11.088090 4084 tablet_copy_service.cc:140] P ad800aa8621b4fc5938692202990b5ab: Received BeginTabletCopySession request for tablet 37f6cffc2bc5456e94999b60340dfeb6 from peer 911714a383074e09a9b2d127ab46d2c1 ({username='slave'} at 127.0.103.69:48253)
I20250627 13:53:11.088371 4084 tablet_copy_service.cc:161] P ad800aa8621b4fc5938692202990b5ab: Beginning new tablet copy session on tablet 37f6cffc2bc5456e94999b60340dfeb6 from peer 911714a383074e09a9b2d127ab46d2c1 at {username='slave'} at 127.0.103.69:48253: session id = 911714a383074e09a9b2d127ab46d2c1-37f6cffc2bc5456e94999b60340dfeb6
I20250627 13:53:11.092418 4084 tablet_copy_source_session.cc:215] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:53:11.094754 5979 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37f6cffc2bc5456e94999b60340dfeb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:11.083616 5953 tablet_copy_client.cc:806] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1: tablet copy: Starting download of 0 data blocks...
I20250627 13:53:11.098212 5953 tablet_copy_client.cc:670] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1: tablet copy: Starting download of 1 WAL segments...
I20250627 13:53:11.102176 5979 tablet_copy_client.cc:806] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1: tablet copy: Starting download of 0 data blocks...
I20250627 13:53:11.102442 5979 tablet_copy_client.cc:670] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1: tablet copy: Starting download of 1 WAL segments...
I20250627 13:53:11.103354 5953 tablet_copy_client.cc:538] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:53:11.104707 5979 tablet_copy_client.cc:538] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:53:11.108822 5979 tablet_bootstrap.cc:492] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1: Bootstrap starting.
I20250627 13:53:11.110157 5727 consensus_queue.cc:1035] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 85, Time since last communication: 0.000s
I20250627 13:53:11.118471 5952 tablet_copy_client.cc:538] T 40b04be88e9947c5a5ca8432e3791bdd P bf0be937c3634bcab62f1e58402a776f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:53:11.120466 5953 tablet_bootstrap.cc:492] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1: Bootstrap starting.
I20250627 13:53:11.122043 5952 tablet_bootstrap.cc:492] T 40b04be88e9947c5a5ca8432e3791bdd P bf0be937c3634bcab62f1e58402a776f: Bootstrap starting.
I20250627 13:53:11.122040 5727 raft_consensus.cc:2953] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab [term 1 LEADER]: Committing config change with OpId 1.87: config changed from index -1 to 87, NON_VOTER 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) added. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: true } } }
I20250627 13:53:11.123230 5957 tablet_bootstrap.cc:492] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c: Bootstrap replayed 1/1 log segments. Stats: ops{read=87 overwritten=0 applied=87 ignored=0} inserts{seen=718 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:53:11.123835 5957 tablet_bootstrap.cc:492] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c: Bootstrap complete.
I20250627 13:53:11.124342 5957 ts_tablet_manager.cc:1397] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c: Time spent bootstrapping tablet: real 0.221s user 0.144s sys 0.014s
I20250627 13:53:11.129523 5957 raft_consensus.cc:357] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: true } }
I20250627 13:53:11.130012 5957 raft_consensus.cc:738] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 36100389495d4971b10cf590bc82c42c, State: Initialized, Role: LEARNER
I20250627 13:53:11.130573 5957 consensus_queue.cc:260] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 87, Last appended: 1.87, Last appended by leader: 87, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: true } }
I20250627 13:53:11.132093 3788 catalog_manager.cc:5582] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab reported cstate change: config changed from index -1 to 87, NON_VOTER 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) added. New cstate: current_term: 1 leader_uuid: "ad800aa8621b4fc5938692202990b5ab" committed_config { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:11.148887 3931 raft_consensus.cc:2953] T 76518bc59dfe443abd8e97539d1bcb8a P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index -1 to 87, NON_VOTER 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) added. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: true } } }
I20250627 13:53:11.207350 5957 ts_tablet_manager.cc:1428] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c: Time spent starting tablet: real 0.083s user 0.035s sys 0.040s
I20250627 13:53:11.209321 4214 tablet_copy_service.cc:342] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Request end of tablet copy session 36100389495d4971b10cf590bc82c42c-8fe13ad746124b1d82cfb8ce691828a4 received from {username='slave'} at 127.0.103.68:36407
I20250627 13:53:11.209498 4214 tablet_copy_service.cc:434] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: ending tablet copy session 36100389495d4971b10cf590bc82c42c-8fe13ad746124b1d82cfb8ce691828a4 on tablet 8fe13ad746124b1d82cfb8ce691828a4 with peer 36100389495d4971b10cf590bc82c42c
I20250627 13:53:11.247138 5953 log.cc:826] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1: Log is configured to *not* fsync() on all Append() calls
I20250627 13:53:11.257284 5972 log.cc:826] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f: Log is configured to *not* fsync() on all Append() calls
W20250627 13:53:11.263829 3997 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.65:35133, user_credentials={real_user=slave}} blocked reactor thread for 100714us
I20250627 13:53:11.361768 5957 ts_tablet_manager.cc:927] T c20692dbde954a259acdb9c765978113 P 36100389495d4971b10cf590bc82c42c: Initiating tablet copy from peer fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67:44627)
I20250627 13:53:11.380343 5957 tablet_copy_client.cc:323] T c20692dbde954a259acdb9c765978113 P 36100389495d4971b10cf590bc82c42c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:44627
I20250627 13:53:11.381147 4214 tablet_copy_service.cc:140] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Received BeginTabletCopySession request for tablet c20692dbde954a259acdb9c765978113 from peer 36100389495d4971b10cf590bc82c42c ({username='slave'} at 127.0.103.68:36407)
I20250627 13:53:11.381321 4214 tablet_copy_service.cc:161] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Beginning new tablet copy session on tablet c20692dbde954a259acdb9c765978113 from peer 36100389495d4971b10cf590bc82c42c at {username='slave'} at 127.0.103.68:36407: session id = 36100389495d4971b10cf590bc82c42c-c20692dbde954a259acdb9c765978113
I20250627 13:53:11.384137 4214 tablet_copy_source_session.cc:215] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:53:11.384680 4194 consensus_queue.cc:237] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 1.87, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: true } }
I20250627 13:53:11.386559 5957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c20692dbde954a259acdb9c765978113. 1 dirs total, 0 dirs full, 0 dirs failed
W20250627 13:53:11.386978 4130 consensus_peers.cc:487] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 -> Peer bf0be937c3634bcab62f1e58402a776f (127.0.103.70:37215): Couldn't send request to peer bf0be937c3634bcab62f1e58402a776f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9f374c0cbcd0493597c8f30e3e63d962. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:53:11.389591 4064 raft_consensus.cc:1273] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 87. Preceding OpId from leader: term: 1 index: 89. (index mismatch)
I20250627 13:53:11.393340 5078 consensus_queue.cc:1035] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 88, Last known committed idx: 87, Time since last communication: 0.000s
I20250627 13:53:11.403189 5957 tablet_copy_client.cc:806] T c20692dbde954a259acdb9c765978113 P 36100389495d4971b10cf590bc82c42c: tablet copy: Starting download of 0 data blocks...
I20250627 13:53:11.406515 5078 raft_consensus.cc:2953] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.88: config changed from index -1 to 88, NON_VOTER bf0be937c3634bcab62f1e58402a776f (127.0.103.70) added. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: true } } }
I20250627 13:53:11.407271 5957 tablet_copy_client.cc:670] T c20692dbde954a259acdb9c765978113 P 36100389495d4971b10cf590bc82c42c: tablet copy: Starting download of 1 WAL segments...
I20250627 13:53:11.409279 5957 tablet_copy_client.cc:538] T c20692dbde954a259acdb9c765978113 P 36100389495d4971b10cf590bc82c42c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:53:11.415783 3788 catalog_manager.cc:5582] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 reported cstate change: config changed from index -1 to 88, NON_VOTER bf0be937c3634bcab62f1e58402a776f (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:11.426798 5957 tablet_bootstrap.cc:492] T c20692dbde954a259acdb9c765978113 P 36100389495d4971b10cf590bc82c42c: Bootstrap starting.
I20250627 13:53:11.441953 4063 raft_consensus.cc:2953] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Committing config change with OpId 1.88: config changed from index -1 to 88, NON_VOTER bf0be937c3634bcab62f1e58402a776f (127.0.103.70) added. New config: { opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: true } } }
I20250627 13:53:11.473515 6001 ts_tablet_manager.cc:927] T 76518bc59dfe443abd8e97539d1bcb8a P 6503b59977444aaebe5ee70a0bc99154: Initiating tablet copy from peer ad800aa8621b4fc5938692202990b5ab (127.0.103.66:43273)
I20250627 13:53:11.490983 4194 consensus_queue.cc:237] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 1.91, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: true } }
W20250627 13:53:11.493088 4129 consensus_peers.cc:487] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 -> Peer 36100389495d4971b10cf590bc82c42c (127.0.103.68:45503): Couldn't send request to peer 36100389495d4971b10cf590bc82c42c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e79731c0edcc4c11a573cf46f0b2c06a. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:53:11.494225 6001 tablet_copy_client.cc:323] T 76518bc59dfe443abd8e97539d1bcb8a P 6503b59977444aaebe5ee70a0bc99154: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:43273
I20250627 13:53:11.498099 3931 raft_consensus.cc:1273] T e79731c0edcc4c11a573cf46f0b2c06a P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 91. Preceding OpId from leader: term: 1 index: 92. (index mismatch)
I20250627 13:53:11.498756 5757 consensus_queue.cc:1035] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20250627 13:53:11.501258 5942 raft_consensus.cc:2953] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.92: config changed from index -1 to 92, NON_VOTER 36100389495d4971b10cf590bc82c42c (127.0.103.68) added. New config: { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: true } } }
I20250627 13:53:11.506774 3933 raft_consensus.cc:2953] T e79731c0edcc4c11a573cf46f0b2c06a P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Committing config change with OpId 1.92: config changed from index -1 to 92, NON_VOTER 36100389495d4971b10cf590bc82c42c (127.0.103.68) added. New config: { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: true } } }
I20250627 13:53:11.509944 3788 catalog_manager.cc:5582] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 reported cstate change: config changed from index -1 to 92, NON_VOTER 36100389495d4971b10cf590bc82c42c (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250627 13:53:11.587764 5565 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.65:35133, user_credentials={real_user=slave}} blocked reactor thread for 46652us
I20250627 13:53:11.617458 4194 consensus_queue.cc:237] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 1.82, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: true } }
W20250627 13:53:11.620478 4128 consensus_peers.cc:487] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 -> Peer 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71:45413): Couldn't send request to peer 6503b59977444aaebe5ee70a0bc99154. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 978e20547d644d3988d6da29235903aa. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:53:11.629227 4060 raft_consensus.cc:1273] T 978e20547d644d3988d6da29235903aa P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 1 index: 83. (index mismatch)
I20250627 13:53:11.631587 5757 consensus_queue.cc:1035] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20250627 13:53:11.631878 4084 tablet_copy_service.cc:140] P ad800aa8621b4fc5938692202990b5ab: Received BeginTabletCopySession request for tablet 76518bc59dfe443abd8e97539d1bcb8a from peer 6503b59977444aaebe5ee70a0bc99154 ({username='slave'} at 127.0.103.71:35855)
I20250627 13:53:11.632047 4084 tablet_copy_service.cc:161] P ad800aa8621b4fc5938692202990b5ab: Beginning new tablet copy session on tablet 76518bc59dfe443abd8e97539d1bcb8a from peer 6503b59977444aaebe5ee70a0bc99154 at {username='slave'} at 127.0.103.71:35855: session id = 6503b59977444aaebe5ee70a0bc99154-76518bc59dfe443abd8e97539d1bcb8a
I20250627 13:53:11.635007 4084 tablet_copy_source_session.cc:215] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:53:11.655721 6001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76518bc59dfe443abd8e97539d1bcb8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:11.657552 5241 raft_consensus.cc:1215] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.78->[1.79-1.102] Dedup: 1.87->[1.88-1.102]
I20250627 13:53:11.665886 6001 tablet_copy_client.cc:806] T 76518bc59dfe443abd8e97539d1bcb8a P 6503b59977444aaebe5ee70a0bc99154: tablet copy: Starting download of 0 data blocks...
I20250627 13:53:11.666260 6001 tablet_copy_client.cc:670] T 76518bc59dfe443abd8e97539d1bcb8a P 6503b59977444aaebe5ee70a0bc99154: tablet copy: Starting download of 1 WAL segments...
I20250627 13:53:11.671213 5952 tablet_bootstrap.cc:492] T 40b04be88e9947c5a5ca8432e3791bdd P bf0be937c3634bcab62f1e58402a776f: Bootstrap replayed 1/1 log segments. Stats: ops{read=74 overwritten=0 applied=74 ignored=0} inserts{seen=746 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:53:11.674294 5940 raft_consensus.cc:2953] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.83: config changed from index -1 to 83, NON_VOTER 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) added. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: true } } }
I20250627 13:53:11.680415 5952 tablet_bootstrap.cc:492] T 40b04be88e9947c5a5ca8432e3791bdd P bf0be937c3634bcab62f1e58402a776f: Bootstrap complete.
I20250627 13:53:11.681110 5952 ts_tablet_manager.cc:1397] T 40b04be88e9947c5a5ca8432e3791bdd P bf0be937c3634bcab62f1e58402a776f: Time spent bootstrapping tablet: real 0.559s user 0.127s sys 0.016s
I20250627 13:53:11.681409 6001 tablet_copy_client.cc:538] T 76518bc59dfe443abd8e97539d1bcb8a P 6503b59977444aaebe5ee70a0bc99154: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:53:11.681617 4060 raft_consensus.cc:2953] T 978e20547d644d3988d6da29235903aa P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index -1 to 83, NON_VOTER 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) added. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: true } } }
I20250627 13:53:11.687891 3788 catalog_manager.cc:5582] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 reported cstate change: config changed from index -1 to 83, NON_VOTER 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) added. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:11.697803 6001 tablet_bootstrap.cc:492] T 76518bc59dfe443abd8e97539d1bcb8a P 6503b59977444aaebe5ee70a0bc99154: Bootstrap starting.
I20250627 13:53:11.703687 5952 raft_consensus.cc:357] T 40b04be88e9947c5a5ca8432e3791bdd P bf0be937c3634bcab62f1e58402a776f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: true } }
I20250627 13:53:11.706797 5952 raft_consensus.cc:738] T 40b04be88e9947c5a5ca8432e3791bdd P bf0be937c3634bcab62f1e58402a776f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bf0be937c3634bcab62f1e58402a776f, State: Initialized, Role: LEARNER
I20250627 13:53:11.713027 5952 consensus_queue.cc:260] T 40b04be88e9947c5a5ca8432e3791bdd P bf0be937c3634bcab62f1e58402a776f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 74, Last appended: 1.74, Last appended by leader: 74, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: true } }
I20250627 13:53:11.726506 5972 tablet_bootstrap.cc:492] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f: Bootstrap replayed 1/1 log segments. Stats: ops{read=97 overwritten=0 applied=97 ignored=0} inserts{seen=728 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:53:11.735616 5972 tablet_bootstrap.cc:492] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f: Bootstrap complete.
I20250627 13:53:11.736260 5972 ts_tablet_manager.cc:1397] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f: Time spent bootstrapping tablet: real 0.682s user 0.161s sys 0.012s
I20250627 13:53:11.747447 5972 raft_consensus.cc:357] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: true } }
I20250627 13:53:11.756623 5972 raft_consensus.cc:738] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bf0be937c3634bcab62f1e58402a776f, State: Initialized, Role: LEARNER
I20250627 13:53:11.757563 5972 consensus_queue.cc:260] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 97, Last appended: 1.97, Last appended by leader: 97, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: true } }
I20250627 13:53:11.760871 5952 ts_tablet_manager.cc:1428] T 40b04be88e9947c5a5ca8432e3791bdd P bf0be937c3634bcab62f1e58402a776f: Time spent starting tablet: real 0.080s user 0.008s sys 0.004s
I20250627 13:53:11.769344 5972 ts_tablet_manager.cc:1428] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f: Time spent starting tablet: real 0.033s user 0.004s sys 0.004s
I20250627 13:53:11.772413 4214 tablet_copy_service.cc:342] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Request end of tablet copy session bf0be937c3634bcab62f1e58402a776f-3e2b11c6788e4ccdbdde60769fee1aae received from {username='slave'} at 127.0.103.70:49887
I20250627 13:53:11.772573 4214 tablet_copy_service.cc:434] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: ending tablet copy session bf0be937c3634bcab62f1e58402a776f-3e2b11c6788e4ccdbdde60769fee1aae on tablet 3e2b11c6788e4ccdbdde60769fee1aae with peer bf0be937c3634bcab62f1e58402a776f
I20250627 13:53:11.773720 3953 tablet_copy_service.cc:342] P f46fd51df5864af9b8493acd69db0747: Request end of tablet copy session bf0be937c3634bcab62f1e58402a776f-40b04be88e9947c5a5ca8432e3791bdd received from {username='slave'} at 127.0.103.70:58947
I20250627 13:53:11.773921 3953 tablet_copy_service.cc:434] P f46fd51df5864af9b8493acd69db0747: ending tablet copy session bf0be937c3634bcab62f1e58402a776f-40b04be88e9947c5a5ca8432e3791bdd on tablet 40b04be88e9947c5a5ca8432e3791bdd with peer bf0be937c3634bcab62f1e58402a776f
I20250627 13:53:11.797518 5967 tablet_bootstrap.cc:492] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154: Bootstrap replayed 1/1 log segments. Stats: ops{read=84 overwritten=0 applied=84 ignored=0} inserts{seen=769 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:53:11.798206 5967 tablet_bootstrap.cc:492] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154: Bootstrap complete.
I20250627 13:53:11.807284 5967 ts_tablet_manager.cc:1397] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154: Time spent bootstrapping tablet: real 0.802s user 0.158s sys 0.022s
I20250627 13:53:11.808719 5967 raft_consensus.cc:357] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: true } }
I20250627 13:53:11.821525 5967 raft_consensus.cc:738] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6503b59977444aaebe5ee70a0bc99154, State: Initialized, Role: LEARNER
I20250627 13:53:11.821929 5967 consensus_queue.cc:260] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 84, Last appended: 1.84, Last appended by leader: 84, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: true } }
I20250627 13:53:11.826892 5967 ts_tablet_manager.cc:1428] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20250627 13:53:11.829137 4214 tablet_copy_service.cc:342] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Request end of tablet copy session 6503b59977444aaebe5ee70a0bc99154-be42f2b9b1b747adb641149f70da608a received from {username='slave'} at 127.0.103.71:57017
I20250627 13:53:11.829316 4214 tablet_copy_service.cc:434] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: ending tablet copy session 6503b59977444aaebe5ee70a0bc99154-be42f2b9b1b747adb641149f70da608a on tablet be42f2b9b1b747adb641149f70da608a with peer 6503b59977444aaebe5ee70a0bc99154
I20250627 13:53:11.833062 5634 consensus_queue.cc:237] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: true } }
I20250627 13:53:11.853143 3931 raft_consensus.cc:1273] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [term 2 FOLLOWER]: Refusing update from remote peer 6503b59977444aaebe5ee70a0bc99154: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250627 13:53:11.854861 6005 consensus_queue.cc:1035] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250627 13:53:11.859844 5502 raft_consensus.cc:1215] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.88->[1.89-1.106] Dedup: 1.97->[1.98-1.106]
W20250627 13:53:11.869778 5567 consensus_peers.cc:487] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 -> Peer 36100389495d4971b10cf590bc82c42c (127.0.103.68:45503): Couldn't send request to peer 36100389495d4971b10cf590bc82c42c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0fdcc98da13c4cf998b1fd1b3e3f9fe5. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:53:11.892985 5953 tablet_bootstrap.cc:492] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=83 overwritten=0 applied=82 ignored=0} inserts{seen=767 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250627 13:53:11.898963 5953 tablet_bootstrap.cc:492] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1: Bootstrap complete.
I20250627 13:53:11.906399 5953 ts_tablet_manager.cc:1397] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1: Time spent bootstrapping tablet: real 0.786s user 0.137s sys 0.021s
I20250627 13:53:11.909390 5940 raft_consensus.cc:1062] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: attempting to promote NON_VOTER 36100389495d4971b10cf590bc82c42c to VOTER
I20250627 13:53:11.910317 5940 consensus_queue.cc:237] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 1.104, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } }
I20250627 13:53:11.911051 6031 ts_tablet_manager.cc:927] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c: Initiating tablet copy from peer fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67:44627)
I20250627 13:53:11.915422 5240 raft_consensus.cc:1273] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c [term 1 LEARNER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch)
I20250627 13:53:11.916635 5757 consensus_queue.cc:1035] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 103, Time since last communication: 0.000s
I20250627 13:53:11.916854 5972 ts_tablet_manager.cc:927] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f: Initiating tablet copy from peer fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67:44627)
I20250627 13:53:11.917279 5953 raft_consensus.cc:357] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: true } }
I20250627 13:53:11.925802 5078 raft_consensus.cc:2953] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.105: config changed from index 79 to 105, 36100389495d4971b10cf590bc82c42c (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } }
I20250627 13:53:11.928378 5239 raft_consensus.cc:2953] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index 79 to 105, 36100389495d4971b10cf590bc82c42c (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } }
I20250627 13:53:11.937891 6031 tablet_copy_client.cc:323] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:44627
I20250627 13:53:11.938732 4214 tablet_copy_service.cc:140] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Received BeginTabletCopySession request for tablet e79731c0edcc4c11a573cf46f0b2c06a from peer 36100389495d4971b10cf590bc82c42c ({username='slave'} at 127.0.103.68:36407)
I20250627 13:53:11.938932 4214 tablet_copy_service.cc:161] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Beginning new tablet copy session on tablet e79731c0edcc4c11a573cf46f0b2c06a from peer 36100389495d4971b10cf590bc82c42c at {username='slave'} at 127.0.103.68:36407: session id = 36100389495d4971b10cf590bc82c42c-e79731c0edcc4c11a573cf46f0b2c06a
I20250627 13:53:11.942139 4214 tablet_copy_source_session.cc:215] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:53:11.942219 4062 raft_consensus.cc:1273] T 8fe13ad746124b1d82cfb8ce691828a4 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch)
I20250627 13:53:11.942866 3789 catalog_manager.cc:5582] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c reported cstate change: config changed from index 79 to 105, 36100389495d4971b10cf590bc82c42c (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } }
I20250627 13:53:11.943622 6031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e79731c0edcc4c11a573cf46f0b2c06a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:11.948032 5953 raft_consensus.cc:738] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 911714a383074e09a9b2d127ab46d2c1, State: Initialized, Role: LEARNER
I20250627 13:53:11.948770 5953 consensus_queue.cc:260] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 82, Last appended: 1.83, Last appended by leader: 83, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: true } }
I20250627 13:53:11.951349 4194 consensus_queue.cc:237] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } }
I20250627 13:53:11.955399 5241 raft_consensus.cc:1273] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch)
I20250627 13:53:11.959086 5078 consensus_queue.cc:1035] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20250627 13:53:11.961067 5972 tablet_copy_client.cc:323] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:44627
I20250627 13:53:11.961880 4214 tablet_copy_service.cc:140] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Received BeginTabletCopySession request for tablet 9f374c0cbcd0493597c8f30e3e63d962 from peer bf0be937c3634bcab62f1e58402a776f ({username='slave'} at 127.0.103.70:49887)
I20250627 13:53:11.962055 4214 tablet_copy_service.cc:161] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Beginning new tablet copy session on tablet 9f374c0cbcd0493597c8f30e3e63d962 from peer bf0be937c3634bcab62f1e58402a776f at {username='slave'} at 127.0.103.70:49887: session id = bf0be937c3634bcab62f1e58402a776f-9f374c0cbcd0493597c8f30e3e63d962
I20250627 13:53:11.965131 4214 tablet_copy_source_session.cc:215] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:53:11.961731 6031 tablet_copy_client.cc:806] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c: tablet copy: Starting download of 0 data blocks...
I20250627 13:53:11.965978 6031 tablet_copy_client.cc:670] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c: tablet copy: Starting download of 1 WAL segments...
I20250627 13:53:11.966059 5850 raft_consensus.cc:2953] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index 105 to 106, VOTER ad800aa8621b4fc5938692202990b5ab (127.0.103.66) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } }
I20250627 13:53:11.966650 5972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f374c0cbcd0493597c8f30e3e63d962. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:11.968497 6031 tablet_copy_client.cc:538] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:53:11.970690 5241 raft_consensus.cc:2953] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index 105 to 106, VOTER ad800aa8621b4fc5938692202990b5ab (127.0.103.66) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } }
I20250627 13:53:11.972316 6031 tablet_bootstrap.cc:492] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c: Bootstrap starting.
I20250627 13:53:11.975754 5953 ts_tablet_manager.cc:1428] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1: Time spent starting tablet: real 0.068s user 0.010s sys 0.001s
I20250627 13:53:11.976660 4214 tablet_copy_service.cc:342] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Request end of tablet copy session 911714a383074e09a9b2d127ab46d2c1-1cd6f75e462546c6bb05ef14af3498fa received from {username='slave'} at 127.0.103.69:36137
I20250627 13:53:11.976816 4214 tablet_copy_service.cc:434] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: ending tablet copy session 911714a383074e09a9b2d127ab46d2c1-1cd6f75e462546c6bb05ef14af3498fa on tablet 1cd6f75e462546c6bb05ef14af3498fa with peer 911714a383074e09a9b2d127ab46d2c1
I20250627 13:53:11.980783 3789 catalog_manager.cc:5582] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c reported cstate change: config changed from index 105 to 106, VOTER ad800aa8621b4fc5938692202990b5ab (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } }
I20250627 13:53:11.982044 5972 tablet_copy_client.cc:806] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f: tablet copy: Starting download of 0 data blocks...
I20250627 13:53:11.990244 3779 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 8fe13ad746124b1d82cfb8ce691828a4 with cas_config_opid_index 105: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:53:11.991111 5972 tablet_copy_client.cc:670] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f: tablet copy: Starting download of 1 WAL segments...
I20250627 13:53:11.993273 5972 tablet_copy_client.cc:538] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:53:11.996742 4042 tablet_service.cc:1515] Processing DeleteTablet for tablet 8fe13ad746124b1d82cfb8ce691828a4 with delete_type TABLET_DATA_TOMBSTONED (TS ad800aa8621b4fc5938692202990b5ab not found in new config with opid_index 106) from {username='slave'} at 127.0.0.1:33136
I20250627 13:53:12.000386 6005 raft_consensus.cc:2953] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 36100389495d4971b10cf590bc82c42c (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: true } } }
I20250627 13:53:12.003623 3930 raft_consensus.cc:2953] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 36100389495d4971b10cf590bc82c42c (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: true } } }
I20250627 13:53:12.009419 3788 catalog_manager.cc:5582] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 reported cstate change: config changed from index -1 to 3, NON_VOTER 36100389495d4971b10cf590bc82c42c (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "6503b59977444aaebe5ee70a0bc99154" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:12.013262 6037 tablet_replica.cc:331] T 8fe13ad746124b1d82cfb8ce691828a4 P ad800aa8621b4fc5938692202990b5ab: stopping tablet replica
I20250627 13:53:12.013947 6037 raft_consensus.cc:2241] T 8fe13ad746124b1d82cfb8ce691828a4 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:53:12.019636 6037 pending_rounds.cc:70] T 8fe13ad746124b1d82cfb8ce691828a4 P ad800aa8621b4fc5938692202990b5ab: Trying to abort 1 pending ops.
I20250627 13:53:12.022475 5372 raft_consensus.cc:1215] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.65->[1.66-1.98] Dedup: 1.83->[1.84-1.98]
I20250627 13:53:12.048215 5501 raft_consensus.cc:1215] T 40b04be88e9947c5a5ca8432e3791bdd P bf0be937c3634bcab62f1e58402a776f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.61->[1.62-1.89] Dedup: 1.74->[1.75-1.89]
I20250627 13:53:12.093814 6037 pending_rounds.cc:77] T 8fe13ad746124b1d82cfb8ce691828a4 P ad800aa8621b4fc5938692202990b5ab: Aborting op as it isn't in flight: id { term: 1 index: 104 } timestamp: 7172228677134434304 op_type: WRITE_OP write_request { tablet_id: "8fe13ad746124b1d82cfb8ce691828a4" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000EU\363]\263\220n\267\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""3465af7cffc8632e" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7172228676955668480 authz_token { token_data: "\010\356\306\372\302\006\"3\n\005slave\022*\n b5d412773a2343d1919352385a3629db\020\001\030\001 \001(\001" signature: "<redacted>""\t\306\223b\374s8;m\224\235\013\307^\234d\021{!\031\253\237q\201\021\022\247\344\'\361\362#\253e1}4\014;\202F72\365\310\336\2445\210\004y\021j}\232\014{\234\354\207I\336\010\201" signing_key_seq_num: 0 } } request_id { client_id: "6df97e60363f402b87cb2656b860393d" seq_no: 260 first_incomplete_seq_no: 260 attempt_no: 0 }
I20250627 13:53:12.108462 6037 raft_consensus.cc:2270] T 8fe13ad746124b1d82cfb8ce691828a4 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:53:12.125543 5979 tablet_bootstrap.cc:492] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=97 overwritten=0 applied=96 ignored=0} inserts{seen=727 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250627 13:53:12.131292 5972 tablet_bootstrap.cc:492] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f: Bootstrap starting.
I20250627 13:53:12.139124 6037 ts_tablet_manager.cc:1905] T 8fe13ad746124b1d82cfb8ce691828a4 P ad800aa8621b4fc5938692202990b5ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:53:12.187042 4194 consensus_queue.cc:237] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 1.109, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: true } }
W20250627 13:53:12.190090 4128 consensus_peers.cc:487] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 -> Peer 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71:45413): Couldn't send request to peer 6503b59977444aaebe5ee70a0bc99154. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a118f20daaf45a2a1c539410dbf0d30. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:53:12.203598 4060 raft_consensus.cc:1273] T 4a118f20daaf45a2a1c539410dbf0d30 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 109. Preceding OpId from leader: term: 1 index: 110. (index mismatch)
I20250627 13:53:12.211223 5979 tablet_bootstrap.cc:492] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1: Bootstrap complete.
I20250627 13:53:12.211589 5979 ts_tablet_manager.cc:1397] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1: Time spent bootstrapping tablet: real 1.103s user 0.159s sys 0.023s
I20250627 13:53:12.220552 5757 consensus_queue.cc:1035] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20250627 13:53:12.224607 5979 raft_consensus.cc:357] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: true } }
I20250627 13:53:12.227926 6001 tablet_bootstrap.cc:492] T 76518bc59dfe443abd8e97539d1bcb8a P 6503b59977444aaebe5ee70a0bc99154: Bootstrap replayed 1/1 log segments. Stats: ops{read=95 overwritten=0 applied=95 ignored=0} inserts{seen=767 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:53:12.228469 6001 tablet_bootstrap.cc:492] T 76518bc59dfe443abd8e97539d1bcb8a P 6503b59977444aaebe5ee70a0bc99154: Bootstrap complete.
I20250627 13:53:12.228832 5979 raft_consensus.cc:738] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 911714a383074e09a9b2d127ab46d2c1, State: Initialized, Role: LEARNER
I20250627 13:53:12.230202 5957 tablet_bootstrap.cc:492] T c20692dbde954a259acdb9c765978113 P 36100389495d4971b10cf590bc82c42c: Bootstrap replayed 1/1 log segments. Stats: ops{read=78 overwritten=0 applied=78 ignored=0} inserts{seen=763 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:53:12.233886 5979 consensus_queue.cc:260] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 96, Last appended: 1.97, Last appended by leader: 97, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: true } }
I20250627 13:53:12.236536 6001 ts_tablet_manager.cc:1397] T 76518bc59dfe443abd8e97539d1bcb8a P 6503b59977444aaebe5ee70a0bc99154: Time spent bootstrapping tablet: real 0.539s user 0.153s sys 0.007s
I20250627 13:53:12.237437 6001 raft_consensus.cc:357] T 76518bc59dfe443abd8e97539d1bcb8a P 6503b59977444aaebe5ee70a0bc99154 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: true } }
I20250627 13:53:12.239557 5979 ts_tablet_manager.cc:1428] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250627 13:53:12.239881 5957 tablet_bootstrap.cc:492] T c20692dbde954a259acdb9c765978113 P 36100389495d4971b10cf590bc82c42c: Bootstrap complete.
I20250627 13:53:12.240063 6001 raft_consensus.cc:738] T 76518bc59dfe443abd8e97539d1bcb8a P 6503b59977444aaebe5ee70a0bc99154 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6503b59977444aaebe5ee70a0bc99154, State: Initialized, Role: LEARNER
I20250627 13:53:12.240324 6001 consensus_queue.cc:260] T 76518bc59dfe443abd8e97539d1bcb8a P 6503b59977444aaebe5ee70a0bc99154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 95, Last appended: 1.95, Last appended by leader: 95, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: true } }
I20250627 13:53:12.240914 5957 ts_tablet_manager.cc:1397] T c20692dbde954a259acdb9c765978113 P 36100389495d4971b10cf590bc82c42c: Time spent bootstrapping tablet: real 0.814s user 0.134s sys 0.011s
I20250627 13:53:12.240605 5967 ts_tablet_manager.cc:927] T 978e20547d644d3988d6da29235903aa P 6503b59977444aaebe5ee70a0bc99154: Initiating tablet copy from peer fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67:44627)
I20250627 13:53:12.243283 4084 tablet_copy_service.cc:342] P ad800aa8621b4fc5938692202990b5ab: Request end of tablet copy session 911714a383074e09a9b2d127ab46d2c1-37f6cffc2bc5456e94999b60340dfeb6 received from {username='slave'} at 127.0.103.69:48253
I20250627 13:53:12.243433 4084 tablet_copy_service.cc:434] P ad800aa8621b4fc5938692202990b5ab: ending tablet copy session 911714a383074e09a9b2d127ab46d2c1-37f6cffc2bc5456e94999b60340dfeb6 on tablet 37f6cffc2bc5456e94999b60340dfeb6 with peer 911714a383074e09a9b2d127ab46d2c1
I20250627 13:53:12.246976 5078 raft_consensus.cc:1062] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6: attempting to promote NON_VOTER bf0be937c3634bcab62f1e58402a776f to VOTER
I20250627 13:53:12.247648 5078 consensus_queue.cc:237] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 113, Committed index: 113, Last appended: 1.114, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } }
I20250627 13:53:12.248428 6001 ts_tablet_manager.cc:1428] T 76518bc59dfe443abd8e97539d1bcb8a P 6503b59977444aaebe5ee70a0bc99154: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250627 13:53:12.250180 5501 raft_consensus.cc:1273] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f [term 1 LEARNER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 113. Preceding OpId from leader: term: 1 index: 115. (index mismatch)
I20250627 13:53:12.250911 5940 consensus_queue.cc:1035] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 112, Time since last communication: 0.000s
I20250627 13:53:12.251567 6037 ts_tablet_manager.cc:1918] T 8fe13ad746124b1d82cfb8ce691828a4 P ad800aa8621b4fc5938692202990b5ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.104
I20250627 13:53:12.251719 6037 log.cc:1199] T 8fe13ad746124b1d82cfb8ce691828a4 P ad800aa8621b4fc5938692202990b5ab: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/wals/8fe13ad746124b1d82cfb8ce691828a4
I20250627 13:53:12.255384 5940 raft_consensus.cc:2953] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.115: config changed from index 89 to 115, bf0be937c3634bcab62f1e58402a776f (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } }
I20250627 13:53:12.259416 5501 raft_consensus.cc:2953] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f [term 1 FOLLOWER]: Committing config change with OpId 1.115: config changed from index 89 to 115, bf0be937c3634bcab62f1e58402a776f (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } }
I20250627 13:53:12.262917 5957 raft_consensus.cc:357] T c20692dbde954a259acdb9c765978113 P 36100389495d4971b10cf590bc82c42c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: true } }
I20250627 13:53:12.265450 5957 raft_consensus.cc:738] T c20692dbde954a259acdb9c765978113 P 36100389495d4971b10cf590bc82c42c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 36100389495d4971b10cf590bc82c42c, State: Initialized, Role: LEARNER
I20250627 13:53:12.266101 5957 consensus_queue.cc:260] T c20692dbde954a259acdb9c765978113 P 36100389495d4971b10cf590bc82c42c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 78, Last appended: 1.78, Last appended by leader: 78, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: true } }
I20250627 13:53:12.267822 5957 ts_tablet_manager.cc:1428] T c20692dbde954a259acdb9c765978113 P 36100389495d4971b10cf590bc82c42c: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20250627 13:53:12.268399 6056 ts_tablet_manager.cc:927] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 36100389495d4971b10cf590bc82c42c: Initiating tablet copy from peer 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71:45413)
I20250627 13:53:12.269763 5967 tablet_copy_client.cc:323] T 978e20547d644d3988d6da29235903aa P 6503b59977444aaebe5ee70a0bc99154: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:44627
W20250627 13:53:12.273586 6042 log.cc:927] Time spent T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20250627 13:53:12.273994 5728 raft_consensus.cc:1062] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747: attempting to promote NON_VOTER bf0be937c3634bcab62f1e58402a776f to VOTER
I20250627 13:53:12.274786 5728 consensus_queue.cc:237] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 1.90, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } }
I20250627 13:53:12.277223 4064 raft_consensus.cc:1273] T 40b04be88e9947c5a5ca8432e3791bdd P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Refusing update from remote peer f46fd51df5864af9b8493acd69db0747: Log matching property violated. Preceding OpId in replica: term: 1 index: 90. Preceding OpId from leader: term: 1 index: 91. (index mismatch)
I20250627 13:53:12.277916 5882 consensus_queue.cc:1035] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20250627 13:53:12.278141 6056 tablet_copy_client.cc:323] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 36100389495d4971b10cf590bc82c42c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.71:45413
I20250627 13:53:12.278587 3933 raft_consensus.cc:1273] T 3e2b11c6788e4ccdbdde60769fee1aae P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 114. Preceding OpId from leader: term: 1 index: 115. (index mismatch)
I20250627 13:53:12.278785 5499 raft_consensus.cc:1273] T 40b04be88e9947c5a5ca8432e3791bdd P bf0be937c3634bcab62f1e58402a776f [term 1 LEARNER]: Refusing update from remote peer f46fd51df5864af9b8493acd69db0747: Log matching property violated. Preceding OpId in replica: term: 1 index: 90. Preceding OpId from leader: term: 1 index: 91. (index mismatch)
I20250627 13:53:12.279645 5882 consensus_queue.cc:1035] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20250627 13:53:12.281991 5882 raft_consensus.cc:2953] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 [term 1 LEADER]: Committing config change with OpId 1.91: config changed from index 62 to 91, bf0be937c3634bcab62f1e58402a776f (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } }
I20250627 13:53:12.282565 4064 raft_consensus.cc:2953] T 40b04be88e9947c5a5ca8432e3791bdd P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Committing config change with OpId 1.91: config changed from index 62 to 91, bf0be937c3634bcab62f1e58402a776f (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } }
I20250627 13:53:12.287554 3789 catalog_manager.cc:5582] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 reported cstate change: config changed from index 62 to 91, bf0be937c3634bcab62f1e58402a776f (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f46fd51df5864af9b8493acd69db0747" committed_config { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:12.292883 5499 raft_consensus.cc:2953] T 40b04be88e9947c5a5ca8432e3791bdd P bf0be937c3634bcab62f1e58402a776f [term 1 FOLLOWER]: Committing config change with OpId 1.91: config changed from index 62 to 91, bf0be937c3634bcab62f1e58402a776f (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } }
I20250627 13:53:12.304627 3777 catalog_manager.cc:4928] TS ad800aa8621b4fc5938692202990b5ab (127.0.103.66:43273): tablet 8fe13ad746124b1d82cfb8ce691828a4 (table rebalance_test_table_4 [id=b5d412773a2343d1919352385a3629db]) successfully deleted
I20250627 13:53:12.306483 5989 raft_consensus.cc:2953] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.110: config changed from index -1 to 110, NON_VOTER 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) added. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: true } } }
I20250627 13:53:12.308019 3932 consensus_queue.cc:237] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 1.91, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } }
I20250627 13:53:12.310650 4084 tablet_copy_service.cc:342] P ad800aa8621b4fc5938692202990b5ab: Request end of tablet copy session 6503b59977444aaebe5ee70a0bc99154-76518bc59dfe443abd8e97539d1bcb8a received from {username='slave'} at 127.0.103.71:35855
I20250627 13:53:12.310787 4084 tablet_copy_service.cc:434] P ad800aa8621b4fc5938692202990b5ab: ending tablet copy session 6503b59977444aaebe5ee70a0bc99154-76518bc59dfe443abd8e97539d1bcb8a on tablet 76518bc59dfe443abd8e97539d1bcb8a with peer 6503b59977444aaebe5ee70a0bc99154
I20250627 13:53:12.310926 4064 raft_consensus.cc:2953] T 4a118f20daaf45a2a1c539410dbf0d30 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Committing config change with OpId 1.110: config changed from index -1 to 110, NON_VOTER 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) added. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: true } } }
I20250627 13:53:12.311230 5499 raft_consensus.cc:1273] T 40b04be88e9947c5a5ca8432e3791bdd P bf0be937c3634bcab62f1e58402a776f [term 1 FOLLOWER]: Refusing update from remote peer f46fd51df5864af9b8493acd69db0747: Log matching property violated. Preceding OpId in replica: term: 1 index: 91. Preceding OpId from leader: term: 1 index: 92. (index mismatch)
I20250627 13:53:12.311738 3789 catalog_manager.cc:5582] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f reported cstate change: config changed from index 89 to 115, bf0be937c3634bcab62f1e58402a776f (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } }
I20250627 13:53:12.311858 5882 consensus_queue.cc:1035] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20250627 13:53:12.318007 5502 raft_consensus.cc:1215] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.113->[1.114-1.114] Dedup: 1.114->[]
I20250627 13:53:12.318090 5728 raft_consensus.cc:2953] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 [term 1 LEADER]: Committing config change with OpId 1.92: config changed from index 91 to 92, VOTER ad800aa8621b4fc5938692202990b5ab (127.0.103.66) evicted. New config: { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } }
I20250627 13:53:12.322362 3776 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 40b04be88e9947c5a5ca8432e3791bdd with cas_config_opid_index 91: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:53:12.323122 5502 raft_consensus.cc:2953] T 40b04be88e9947c5a5ca8432e3791bdd P bf0be937c3634bcab62f1e58402a776f [term 1 FOLLOWER]: Committing config change with OpId 1.92: config changed from index 91 to 92, VOTER ad800aa8621b4fc5938692202990b5ab (127.0.103.66) evicted. New config: { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } }
I20250627 13:53:12.329874 3789 catalog_manager.cc:5582] T 40b04be88e9947c5a5ca8432e3791bdd P f46fd51df5864af9b8493acd69db0747 reported cstate change: config changed from index 91 to 92, VOTER ad800aa8621b4fc5938692202990b5ab (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "f46fd51df5864af9b8493acd69db0747" committed_config { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:53:12.333678 3788 catalog_manager.cc:5582] T 4a118f20daaf45a2a1c539410dbf0d30 P ad800aa8621b4fc5938692202990b5ab reported cstate change: config changed from index -1 to 110, NON_VOTER 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) added. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: true } } }
I20250627 13:53:12.342339 4214 tablet_copy_service.cc:342] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Request end of tablet copy session 36100389495d4971b10cf590bc82c42c-c20692dbde954a259acdb9c765978113 received from {username='slave'} at 127.0.103.68:36407
I20250627 13:53:12.343451 4042 tablet_service.cc:1515] Processing DeleteTablet for tablet 40b04be88e9947c5a5ca8432e3791bdd with delete_type TABLET_DATA_TOMBSTONED (TS ad800aa8621b4fc5938692202990b5ab not found in new config with opid_index 92) from {username='slave'} at 127.0.0.1:33136
I20250627 13:53:12.344743 6037 tablet_replica.cc:331] T 40b04be88e9947c5a5ca8432e3791bdd P ad800aa8621b4fc5938692202990b5ab: stopping tablet replica
I20250627 13:53:12.344959 6037 raft_consensus.cc:2241] T 40b04be88e9947c5a5ca8432e3791bdd P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:53:12.345201 6037 raft_consensus.cc:2270] T 40b04be88e9947c5a5ca8432e3791bdd P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:53:12.346436 6037 ts_tablet_manager.cc:1905] T 40b04be88e9947c5a5ca8432e3791bdd P ad800aa8621b4fc5938692202990b5ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:53:12.349152 5655 tablet_copy_service.cc:140] P 6503b59977444aaebe5ee70a0bc99154: Received BeginTabletCopySession request for tablet 0fdcc98da13c4cf998b1fd1b3e3f9fe5 from peer 36100389495d4971b10cf590bc82c42c ({username='slave'} at 127.0.103.68:51275)
I20250627 13:53:12.349380 5655 tablet_copy_service.cc:161] P 6503b59977444aaebe5ee70a0bc99154: Beginning new tablet copy session on tablet 0fdcc98da13c4cf998b1fd1b3e3f9fe5 from peer 36100389495d4971b10cf590bc82c42c at {username='slave'} at 127.0.103.68:51275: session id = 36100389495d4971b10cf590bc82c42c-0fdcc98da13c4cf998b1fd1b3e3f9fe5
I20250627 13:53:12.352905 5655 tablet_copy_source_session.cc:215] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:53:12.353853 6037 ts_tablet_manager.cc:1918] T 40b04be88e9947c5a5ca8432e3791bdd P ad800aa8621b4fc5938692202990b5ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.91
I20250627 13:53:12.354071 6037 log.cc:1199] T 40b04be88e9947c5a5ca8432e3791bdd P ad800aa8621b4fc5938692202990b5ab: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/wals/40b04be88e9947c5a5ca8432e3791bdd
I20250627 13:53:12.354641 6056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fdcc98da13c4cf998b1fd1b3e3f9fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:12.354975 3777 catalog_manager.cc:4928] TS ad800aa8621b4fc5938692202990b5ab (127.0.103.66:43273): tablet 40b04be88e9947c5a5ca8432e3791bdd (table rebalance_test_table_2 [id=41b84e234b64410a9ea4c057e1c398cd]) successfully deleted
I20250627 13:53:12.356436 5078 consensus_queue.cc:1035] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 113, Time since last communication: 0.076s
I20250627 13:53:12.360224 4213 tablet_copy_service.cc:140] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Received BeginTabletCopySession request for tablet 978e20547d644d3988d6da29235903aa from peer 6503b59977444aaebe5ee70a0bc99154 ({username='slave'} at 127.0.103.71:57017)
I20250627 13:53:12.361583 6056 tablet_copy_client.cc:806] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 36100389495d4971b10cf590bc82c42c: tablet copy: Starting download of 0 data blocks...
I20250627 13:53:12.361873 6056 tablet_copy_client.cc:670] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 36100389495d4971b10cf590bc82c42c: tablet copy: Starting download of 1 WAL segments...
I20250627 13:53:12.363626 4214 tablet_copy_service.cc:434] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: ending tablet copy session 36100389495d4971b10cf590bc82c42c-c20692dbde954a259acdb9c765978113 on tablet c20692dbde954a259acdb9c765978113 with peer 36100389495d4971b10cf590bc82c42c
I20250627 13:53:12.367311 6056 tablet_copy_client.cc:538] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 36100389495d4971b10cf590bc82c42c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:53:12.366000 4213 tablet_copy_service.cc:161] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Beginning new tablet copy session on tablet 978e20547d644d3988d6da29235903aa from peer 6503b59977444aaebe5ee70a0bc99154 at {username='slave'} at 127.0.103.71:57017: session id = 6503b59977444aaebe5ee70a0bc99154-978e20547d644d3988d6da29235903aa
I20250627 13:53:12.372308 5078 raft_consensus.cc:1062] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6: attempting to promote NON_VOTER 911714a383074e09a9b2d127ab46d2c1 to VOTER
I20250627 13:53:12.377110 6056 tablet_bootstrap.cc:492] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 36100389495d4971b10cf590bc82c42c: Bootstrap starting.
I20250627 13:53:12.391122 4213 tablet_copy_source_session.cc:215] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:53:12.392956 5967 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 978e20547d644d3988d6da29235903aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:12.395396 6056 tablet_bootstrap.cc:492] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 36100389495d4971b10cf590bc82c42c: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:53:12.395896 6056 tablet_bootstrap.cc:492] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 36100389495d4971b10cf590bc82c42c: Bootstrap complete.
I20250627 13:53:12.396202 6056 ts_tablet_manager.cc:1397] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 36100389495d4971b10cf590bc82c42c: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.007s
I20250627 13:53:12.397073 6056 raft_consensus.cc:357] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 36100389495d4971b10cf590bc82c42c [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: true } }
I20250627 13:53:12.399195 5967 tablet_copy_client.cc:806] T 978e20547d644d3988d6da29235903aa P 6503b59977444aaebe5ee70a0bc99154: tablet copy: Starting download of 0 data blocks...
I20250627 13:53:12.399454 5967 tablet_copy_client.cc:670] T 978e20547d644d3988d6da29235903aa P 6503b59977444aaebe5ee70a0bc99154: tablet copy: Starting download of 1 WAL segments...
I20250627 13:53:12.401434 5967 tablet_copy_client.cc:538] T 978e20547d644d3988d6da29235903aa P 6503b59977444aaebe5ee70a0bc99154: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:53:12.405007 5967 tablet_bootstrap.cc:492] T 978e20547d644d3988d6da29235903aa P 6503b59977444aaebe5ee70a0bc99154: Bootstrap starting.
W20250627 13:53:12.407661 4127 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.65:35133, user_credentials={real_user=slave}} blocked reactor thread for 104432us
I20250627 13:53:12.411160 6056 raft_consensus.cc:738] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 36100389495d4971b10cf590bc82c42c [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 36100389495d4971b10cf590bc82c42c, State: Initialized, Role: LEARNER
I20250627 13:53:12.411676 6056 consensus_queue.cc:260] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 36100389495d4971b10cf590bc82c42c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: true } }
I20250627 13:53:12.412173 3931 raft_consensus.cc:2953] T 3e2b11c6788e4ccdbdde60769fee1aae P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Committing config change with OpId 1.115: config changed from index 89 to 115, bf0be937c3634bcab62f1e58402a776f (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } }
I20250627 13:53:12.414830 6056 ts_tablet_manager.cc:1428] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 36100389495d4971b10cf590bc82c42c: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20250627 13:53:12.417799 5655 tablet_copy_service.cc:342] P 6503b59977444aaebe5ee70a0bc99154: Request end of tablet copy session 36100389495d4971b10cf590bc82c42c-0fdcc98da13c4cf998b1fd1b3e3f9fe5 received from {username='slave'} at 127.0.103.68:51275
I20250627 13:53:12.418025 5655 tablet_copy_service.cc:434] P 6503b59977444aaebe5ee70a0bc99154: ending tablet copy session 36100389495d4971b10cf590bc82c42c-0fdcc98da13c4cf998b1fd1b3e3f9fe5 on tablet 0fdcc98da13c4cf998b1fd1b3e3f9fe5 with peer 36100389495d4971b10cf590bc82c42c
W20250627 13:53:12.437809 4130 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.70:37215, user_credentials={real_user=slave}} blocked reactor thread for 82266.4us
I20250627 13:53:12.438750 5078 consensus_queue.cc:237] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 1.101, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } }
I20250627 13:53:12.440451 5241 raft_consensus.cc:1215] T c20692dbde954a259acdb9c765978113 P 36100389495d4971b10cf590bc82c42c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.70->[1.71-1.82] Dedup: 1.78->[1.79-1.82]
I20250627 13:53:12.440807 3931 raft_consensus.cc:1273] T 1cd6f75e462546c6bb05ef14af3498fa P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 100. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20250627 13:53:12.441344 5940 consensus_queue.cc:1035] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 100, Time since last communication: 0.000s
I20250627 13:53:12.447752 5940 raft_consensus.cc:2953] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.102: config changed from index 66 to 102, 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } }
I20250627 13:53:12.460367 3931 raft_consensus.cc:2953] T 1cd6f75e462546c6bb05ef14af3498fa P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index 66 to 102, 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } }
I20250627 13:53:12.463819 3788 catalog_manager.cc:5582] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 reported cstate change: config changed from index 66 to 102, 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250627 13:53:12.488862 4128 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.66:43273, user_credentials={real_user=slave}} blocked reactor thread for 150243us
I20250627 13:53:12.491243 5634 raft_consensus.cc:1215] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.73->[1.74-1.110] Dedup: 1.84->[1.85-1.110]
I20250627 13:53:12.497182 5633 raft_consensus.cc:1215] T 76518bc59dfe443abd8e97539d1bcb8a P 6503b59977444aaebe5ee70a0bc99154 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.86->[1.87-1.111] Dedup: 1.95->[1.96-1.111]
I20250627 13:53:12.499465 5372 raft_consensus.cc:1273] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1 [term 1 LEARNER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 101. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20250627 13:53:12.509289 5942 consensus_queue.cc:1035] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 100, Time since last communication: 0.000s
I20250627 13:53:12.521939 5372 raft_consensus.cc:2953] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1 [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index 66 to 102, 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } }
I20250627 13:53:12.537681 6031 tablet_bootstrap.cc:492] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c: Bootstrap replayed 1/1 log segments. Stats: ops{read=99 overwritten=0 applied=99 ignored=0} inserts{seen=730 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:53:12.538636 6031 tablet_bootstrap.cc:492] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c: Bootstrap complete.
I20250627 13:53:12.539062 6031 ts_tablet_manager.cc:1397] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c: Time spent bootstrapping tablet: real 0.567s user 0.182s sys 0.015s
I20250627 13:53:12.539918 6031 raft_consensus.cc:357] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: true } }
I20250627 13:53:12.540233 6031 raft_consensus.cc:738] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 36100389495d4971b10cf590bc82c42c, State: Initialized, Role: LEARNER
I20250627 13:53:12.540496 6031 consensus_queue.cc:260] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 99, Last appended: 1.99, Last appended by leader: 99, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: true } }
I20250627 13:53:12.543593 6031 ts_tablet_manager.cc:1428] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250627 13:53:12.546627 4213 tablet_copy_service.cc:342] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Request end of tablet copy session 36100389495d4971b10cf590bc82c42c-e79731c0edcc4c11a573cf46f0b2c06a received from {username='slave'} at 127.0.103.68:36407
I20250627 13:53:12.546800 4213 tablet_copy_service.cc:434] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: ending tablet copy session 36100389495d4971b10cf590bc82c42c-e79731c0edcc4c11a573cf46f0b2c06a on tablet e79731c0edcc4c11a573cf46f0b2c06a with peer 36100389495d4971b10cf590bc82c42c
I20250627 13:53:12.619813 5372 raft_consensus.cc:1215] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.91->[1.92-1.108] Dedup: 1.97->[1.98-1.108]
I20250627 13:53:12.641348 6065 raft_consensus.cc:1062] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: attempting to promote NON_VOTER 36100389495d4971b10cf590bc82c42c to VOTER
I20250627 13:53:12.642261 6065 consensus_queue.cc:237] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 1.83, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } }
I20250627 13:53:12.645777 5241 raft_consensus.cc:1273] T c20692dbde954a259acdb9c765978113 P 36100389495d4971b10cf590bc82c42c [term 1 LEARNER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20250627 13:53:12.647352 6063 consensus_queue.cc:1035] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 82, Time since last communication: 0.000s
I20250627 13:53:12.653697 6065 raft_consensus.cc:2953] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.84: config changed from index 71 to 84, 36100389495d4971b10cf590bc82c42c (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } }
I20250627 13:53:12.658594 5241 raft_consensus.cc:2953] T c20692dbde954a259acdb9c765978113 P 36100389495d4971b10cf590bc82c42c [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index 71 to 84, 36100389495d4971b10cf590bc82c42c (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } }
I20250627 13:53:12.661289 3789 catalog_manager.cc:5582] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 reported cstate change: config changed from index 71 to 84, 36100389495d4971b10cf590bc82c42c (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:53:12.691464 5241 raft_consensus.cc:1215] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 36100389495d4971b10cf590bc82c42c [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250627 13:53:12.772377 4194 consensus_queue.cc:237] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 1.85, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } }
I20250627 13:53:12.774649 5241 raft_consensus.cc:1273] T c20692dbde954a259acdb9c765978113 P 36100389495d4971b10cf590bc82c42c [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 85. Preceding OpId from leader: term: 1 index: 86. (index mismatch)
I20250627 13:53:12.775697 6063 consensus_queue.cc:1035] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20250627 13:53:12.783200 5942 raft_consensus.cc:2953] T c20692dbde954a259acdb9c765978113 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.86: config changed from index 84 to 86, VOTER ad800aa8621b4fc5938692202990b5ab (127.0.103.66) evicted. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } }
I20250627 13:53:12.789837 5241 raft_consensus.cc:2953] T c20692dbde954a259acdb9c765978113 P 36100389495d4971b10cf590bc82c42c [term 1 FOLLOWER]: Committing config change with OpId 1.86: config changed from index 84 to 86, VOTER ad800aa8621b4fc5938692202990b5ab (127.0.103.66) evicted. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } }
I20250627 13:53:12.797621 3784 catalog_manager.cc:5582] T c20692dbde954a259acdb9c765978113 P 36100389495d4971b10cf590bc82c42c reported cstate change: config changed from index 84 to 86, VOTER ad800aa8621b4fc5938692202990b5ab (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } }
I20250627 13:53:12.802187 4042 tablet_service.cc:1515] Processing DeleteTablet for tablet c20692dbde954a259acdb9c765978113 with delete_type TABLET_DATA_TOMBSTONED (TS ad800aa8621b4fc5938692202990b5ab not found in new config with opid_index 86) from {username='slave'} at 127.0.0.1:33136
I20250627 13:53:12.803202 6037 tablet_replica.cc:331] T c20692dbde954a259acdb9c765978113 P ad800aa8621b4fc5938692202990b5ab: stopping tablet replica
I20250627 13:53:12.803545 6037 raft_consensus.cc:2241] T c20692dbde954a259acdb9c765978113 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:53:12.804225 6037 raft_consensus.cc:2270] T c20692dbde954a259acdb9c765978113 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:53:12.805676 6037 ts_tablet_manager.cc:1905] T c20692dbde954a259acdb9c765978113 P ad800aa8621b4fc5938692202990b5ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:53:12.841936 3779 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet c20692dbde954a259acdb9c765978113 with cas_config_opid_index 84: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20250627 13:53:12.844936 4128 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.66:43273, user_credentials={real_user=slave}} blocked reactor thread for 218787us
I20250627 13:53:12.858580 6083 ts_tablet_manager.cc:927] T 4a118f20daaf45a2a1c539410dbf0d30 P 6503b59977444aaebe5ee70a0bc99154: Initiating tablet copy from peer fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67:44627)
I20250627 13:53:12.876988 6083 tablet_copy_client.cc:323] T 4a118f20daaf45a2a1c539410dbf0d30 P 6503b59977444aaebe5ee70a0bc99154: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:44627
I20250627 13:53:12.879279 4213 tablet_copy_service.cc:140] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Received BeginTabletCopySession request for tablet 4a118f20daaf45a2a1c539410dbf0d30 from peer 6503b59977444aaebe5ee70a0bc99154 ({username='slave'} at 127.0.103.71:57017)
I20250627 13:53:12.879468 4213 tablet_copy_service.cc:161] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Beginning new tablet copy session on tablet 4a118f20daaf45a2a1c539410dbf0d30 from peer 6503b59977444aaebe5ee70a0bc99154 at {username='slave'} at 127.0.103.71:57017: session id = 6503b59977444aaebe5ee70a0bc99154-4a118f20daaf45a2a1c539410dbf0d30
I20250627 13:53:12.899235 5727 raft_consensus.cc:1062] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab: attempting to promote NON_VOTER 911714a383074e09a9b2d127ab46d2c1 to VOTER
I20250627 13:53:12.900120 5727 consensus_queue.cc:237] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 1.109, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } }
I20250627 13:53:12.906719 5372 raft_consensus.cc:1273] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1 [term 1 LEARNER]: Refusing update from remote peer ad800aa8621b4fc5938692202990b5ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 109. Preceding OpId from leader: term: 1 index: 110. (index mismatch)
I20250627 13:53:12.907442 5985 consensus_queue.cc:1035] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 108, Time since last communication: 0.000s
I20250627 13:53:12.912950 5985 raft_consensus.cc:2953] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [term 1 LEADER]: Committing config change with OpId 1.110: config changed from index 92 to 110, 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } }
I20250627 13:53:12.921005 5372 raft_consensus.cc:2953] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1 [term 1 FOLLOWER]: Committing config change with OpId 1.110: config changed from index 92 to 110, 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } }
I20250627 13:53:12.934033 3784 catalog_manager.cc:5582] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1 reported cstate change: config changed from index 92 to 110, 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ad800aa8621b4fc5938692202990b5ab" committed_config { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } }
I20250627 13:53:12.950362 4064 consensus_queue.cc:237] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 1.110, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } }
I20250627 13:53:12.951977 5241 raft_consensus.cc:1215] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.91->[1.92-1.114] Dedup: 1.99->[1.100-1.114]
I20250627 13:53:12.955963 5967 tablet_bootstrap.cc:492] T 978e20547d644d3988d6da29235903aa P 6503b59977444aaebe5ee70a0bc99154: Bootstrap replayed 1/1 log segments. Stats: ops{read=90 overwritten=0 applied=90 ignored=0} inserts{seen=775 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:53:12.956488 5967 tablet_bootstrap.cc:492] T 978e20547d644d3988d6da29235903aa P 6503b59977444aaebe5ee70a0bc99154: Bootstrap complete.
I20250627 13:53:12.956772 5967 ts_tablet_manager.cc:1397] T 978e20547d644d3988d6da29235903aa P 6503b59977444aaebe5ee70a0bc99154: Time spent bootstrapping tablet: real 0.552s user 0.184s sys 0.012s
I20250627 13:53:12.957629 5967 raft_consensus.cc:357] T 978e20547d644d3988d6da29235903aa P 6503b59977444aaebe5ee70a0bc99154 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: true } }
I20250627 13:53:12.957859 5967 raft_consensus.cc:738] T 978e20547d644d3988d6da29235903aa P 6503b59977444aaebe5ee70a0bc99154 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6503b59977444aaebe5ee70a0bc99154, State: Initialized, Role: LEARNER
I20250627 13:53:12.959933 4213 tablet_copy_source_session.cc:215] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:53:12.962649 6037 ts_tablet_manager.cc:1918] T c20692dbde954a259acdb9c765978113 P ad800aa8621b4fc5938692202990b5ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.82
I20250627 13:53:12.962805 6037 log.cc:1199] T c20692dbde954a259acdb9c765978113 P ad800aa8621b4fc5938692202990b5ab: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/wals/c20692dbde954a259acdb9c765978113
I20250627 13:53:12.965019 3777 catalog_manager.cc:4928] TS ad800aa8621b4fc5938692202990b5ab (127.0.103.66:43273): tablet c20692dbde954a259acdb9c765978113 (table rebalance_test_table_0 [id=59a8f27f17c04730859e43c3fde21ee6]) successfully deleted
I20250627 13:53:12.965317 5967 consensus_queue.cc:260] T 978e20547d644d3988d6da29235903aa P 6503b59977444aaebe5ee70a0bc99154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 90, Last appended: 1.90, Last appended by leader: 90, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: true } }
I20250627 13:53:12.966492 5967 ts_tablet_manager.cc:1428] T 978e20547d644d3988d6da29235903aa P 6503b59977444aaebe5ee70a0bc99154: Time spent starting tablet: real 0.010s user 0.000s sys 0.003s
I20250627 13:53:12.978538 4194 consensus_queue.cc:237] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 1.106, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } }
I20250627 13:53:12.980122 4213 tablet_copy_service.cc:342] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Request end of tablet copy session 6503b59977444aaebe5ee70a0bc99154-978e20547d644d3988d6da29235903aa received from {username='slave'} at 127.0.103.71:57017
I20250627 13:53:12.980324 4213 tablet_copy_service.cc:434] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: ending tablet copy session 6503b59977444aaebe5ee70a0bc99154-978e20547d644d3988d6da29235903aa on tablet 978e20547d644d3988d6da29235903aa with peer 6503b59977444aaebe5ee70a0bc99154
I20250627 13:53:12.980683 6083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a118f20daaf45a2a1c539410dbf0d30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:12.988555 5372 raft_consensus.cc:1273] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1 [term 1 FOLLOWER]: Refusing update from remote peer ad800aa8621b4fc5938692202990b5ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 110. Preceding OpId from leader: term: 1 index: 111. (index mismatch)
I20250627 13:53:12.989919 5727 consensus_queue.cc:1035] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20250627 13:53:12.990036 6083 tablet_copy_client.cc:806] T 4a118f20daaf45a2a1c539410dbf0d30 P 6503b59977444aaebe5ee70a0bc99154: tablet copy: Starting download of 0 data blocks...
I20250627 13:53:12.990396 6083 tablet_copy_client.cc:670] T 4a118f20daaf45a2a1c539410dbf0d30 P 6503b59977444aaebe5ee70a0bc99154: tablet copy: Starting download of 1 WAL segments...
I20250627 13:53:12.992375 6083 tablet_copy_client.cc:538] T 4a118f20daaf45a2a1c539410dbf0d30 P 6503b59977444aaebe5ee70a0bc99154: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:53:12.992950 5760 raft_consensus.cc:2953] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [term 1 LEADER]: Committing config change with OpId 1.111: config changed from index 110 to 111, VOTER fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67) evicted. New config: { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } }
I20250627 13:53:12.994966 5989 raft_consensus.cc:1062] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6: attempting to promote NON_VOTER 36100389495d4971b10cf590bc82c42c to VOTER
I20250627 13:53:12.995712 5989 consensus_queue.cc:237] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 1.115, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } }
I20250627 13:53:12.997915 3777 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 37f6cffc2bc5456e94999b60340dfeb6 with cas_config_opid_index 110: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:53:12.998601 3931 raft_consensus.cc:1273] T e79731c0edcc4c11a573cf46f0b2c06a P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 115. Preceding OpId from leader: term: 1 index: 116. (index mismatch)
I20250627 13:53:12.998622 5240 raft_consensus.cc:1273] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c [term 1 LEARNER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 115. Preceding OpId from leader: term: 1 index: 116. (index mismatch)
I20250627 13:53:12.998708 5371 raft_consensus.cc:2953] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1 [term 1 FOLLOWER]: Committing config change with OpId 1.111: config changed from index 110 to 111, VOTER fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67) evicted. New config: { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } }
I20250627 13:53:12.999100 6063 consensus_queue.cc:1035] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20250627 13:53:13.000098 6063 consensus_queue.cc:1035] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 116, Last known committed idx: 115, Time since last communication: 0.000s
I20250627 13:53:13.002563 6083 tablet_bootstrap.cc:492] T 4a118f20daaf45a2a1c539410dbf0d30 P 6503b59977444aaebe5ee70a0bc99154: Bootstrap starting.
I20250627 13:53:13.002851 6060 raft_consensus.cc:2953] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.116: config changed from index 92 to 116, 36100389495d4971b10cf590bc82c42c (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } }
I20250627 13:53:13.006251 5241 raft_consensus.cc:2953] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c [term 1 FOLLOWER]: Committing config change with OpId 1.116: config changed from index 92 to 116, 36100389495d4971b10cf590bc82c42c (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } }
I20250627 13:53:13.008852 3930 raft_consensus.cc:2953] T e79731c0edcc4c11a573cf46f0b2c06a P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Committing config change with OpId 1.116: config changed from index 92 to 116, 36100389495d4971b10cf590bc82c42c (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } }
I20250627 13:53:13.012511 3789 catalog_manager.cc:5582] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c reported cstate change: config changed from index 92 to 116, 36100389495d4971b10cf590bc82c42c (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 116 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } }
I20250627 13:53:13.019958 3784 catalog_manager.cc:5582] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1 reported cstate change: config changed from index 110 to 111, VOTER fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67) evicted. New cstate: current_term: 1 leader_uuid: "ad800aa8621b4fc5938692202990b5ab" committed_config { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } }
I20250627 13:53:13.036609 4174 tablet_service.cc:1515] Processing DeleteTablet for tablet 37f6cffc2bc5456e94999b60340dfeb6 with delete_type TABLET_DATA_TOMBSTONED (TS fb586b8cb7ab44e1a7ec5016dbd2c0a6 not found in new config with opid_index 111) from {username='slave'} at 127.0.0.1:32882
I20250627 13:53:13.053151 4192 raft_consensus.cc:1273] T 37f6cffc2bc5456e94999b60340dfeb6 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Refusing update from remote peer ad800aa8621b4fc5938692202990b5ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 109. Preceding OpId from leader: term: 1 index: 110. (index mismatch)
I20250627 13:53:13.055847 5372 raft_consensus.cc:1273] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1 [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 106. Preceding OpId from leader: term: 1 index: 107. (index mismatch)
I20250627 13:53:13.060827 4194 consensus_queue.cc:237] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 1.117, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } }
I20250627 13:53:13.064306 5240 raft_consensus.cc:1273] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 117. Preceding OpId from leader: term: 1 index: 118. (index mismatch)
I20250627 13:53:13.066099 5989 consensus_queue.cc:1035] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 117, Time since last communication: 0.000s
I20250627 13:53:13.071159 6066 raft_consensus.cc:2953] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.118: config changed from index 116 to 118, VOTER f46fd51df5864af9b8493acd69db0747 (127.0.103.65) evicted. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } }
I20250627 13:53:13.074509 5241 raft_consensus.cc:2953] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c [term 1 FOLLOWER]: Committing config change with OpId 1.118: config changed from index 116 to 118, VOTER f46fd51df5864af9b8493acd69db0747 (127.0.103.65) evicted. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } }
I20250627 13:53:13.085006 3779 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet e79731c0edcc4c11a573cf46f0b2c06a with cas_config_opid_index 116: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:53:13.088132 3789 catalog_manager.cc:5582] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 reported cstate change: config changed from index 116 to 118, VOTER f46fd51df5864af9b8493acd69db0747 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:53:13.092901 3912 tablet_service.cc:1515] Processing DeleteTablet for tablet e79731c0edcc4c11a573cf46f0b2c06a with delete_type TABLET_DATA_TOMBSTONED (TS f46fd51df5864af9b8493acd69db0747 not found in new config with opid_index 118) from {username='slave'} at 127.0.0.1:41436
I20250627 13:53:13.095722 6097 tablet_replica.cc:331] T e79731c0edcc4c11a573cf46f0b2c06a P f46fd51df5864af9b8493acd69db0747: stopping tablet replica
I20250627 13:53:13.096182 6097 raft_consensus.cc:2241] T e79731c0edcc4c11a573cf46f0b2c06a P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:53:13.109879 6097 raft_consensus.cc:2270] T e79731c0edcc4c11a573cf46f0b2c06a P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:53:13.116688 6097 ts_tablet_manager.cc:1905] T e79731c0edcc4c11a573cf46f0b2c06a P f46fd51df5864af9b8493acd69db0747: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:53:13.124104 6060 consensus_queue.cc:1035] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.010s
I20250627 13:53:13.130851 6094 raft_consensus.cc:1062] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab: attempting to promote NON_VOTER 6503b59977444aaebe5ee70a0bc99154 to VOTER
I20250627 13:53:13.131686 6094 consensus_queue.cc:237] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 1.120, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } }
I20250627 13:53:13.134431 3930 raft_consensus.cc:1273] T 76518bc59dfe443abd8e97539d1bcb8a P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Refusing update from remote peer ad800aa8621b4fc5938692202990b5ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 120. Preceding OpId from leader: term: 1 index: 121. (index mismatch)
I20250627 13:53:13.135195 5760 consensus_queue.cc:1035] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20250627 13:53:13.138453 5760 raft_consensus.cc:2953] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab [term 1 LEADER]: Committing config change with OpId 1.121: config changed from index 87 to 121, 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } } }
I20250627 13:53:13.142055 5633 raft_consensus.cc:1273] T 76518bc59dfe443abd8e97539d1bcb8a P 6503b59977444aaebe5ee70a0bc99154 [term 1 LEARNER]: Refusing update from remote peer ad800aa8621b4fc5938692202990b5ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 120. Preceding OpId from leader: term: 1 index: 121. (index mismatch)
I20250627 13:53:13.146034 5727 consensus_queue.cc:1035] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20250627 13:53:13.148540 5633 raft_consensus.cc:2953] T 76518bc59dfe443abd8e97539d1bcb8a P 6503b59977444aaebe5ee70a0bc99154 [term 1 FOLLOWER]: Committing config change with OpId 1.121: config changed from index 87 to 121, 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } } }
I20250627 13:53:13.149607 3930 raft_consensus.cc:2953] T 76518bc59dfe443abd8e97539d1bcb8a P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Committing config change with OpId 1.121: config changed from index 87 to 121, 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } } }
I20250627 13:53:13.154887 3784 catalog_manager.cc:5582] T 76518bc59dfe443abd8e97539d1bcb8a P 6503b59977444aaebe5ee70a0bc99154 reported cstate change: config changed from index 87 to 121, 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ad800aa8621b4fc5938692202990b5ab" committed_config { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } } }
I20250627 13:53:13.164587 5372 raft_consensus.cc:2953] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1 [term 1 FOLLOWER]: Committing config change with OpId 1.107: config changed from index 102 to 107, VOTER f46fd51df5864af9b8493acd69db0747 (127.0.103.65) evicted. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } }
I20250627 13:53:13.170332 6060 raft_consensus.cc:2953] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.107: config changed from index 102 to 107, VOTER f46fd51df5864af9b8493acd69db0747 (127.0.103.65) evicted. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } }
I20250627 13:53:13.175388 4062 consensus_queue.cc:237] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 1.121, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } }
I20250627 13:53:13.177342 6095 tablet_replica.cc:331] T 37f6cffc2bc5456e94999b60340dfeb6 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: stopping tablet replica
I20250627 13:53:13.181232 6066 raft_consensus.cc:1062] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6: attempting to promote NON_VOTER 6503b59977444aaebe5ee70a0bc99154 to VOTER
I20250627 13:53:13.182974 6095 raft_consensus.cc:2241] T 37f6cffc2bc5456e94999b60340dfeb6 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:53:13.183252 5633 raft_consensus.cc:1273] T 76518bc59dfe443abd8e97539d1bcb8a P 6503b59977444aaebe5ee70a0bc99154 [term 1 FOLLOWER]: Refusing update from remote peer ad800aa8621b4fc5938692202990b5ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 121. Preceding OpId from leader: term: 1 index: 122. (index mismatch)
I20250627 13:53:13.183915 6094 consensus_queue.cc:1035] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20250627 13:53:13.184248 6095 pending_rounds.cc:70] T 37f6cffc2bc5456e94999b60340dfeb6 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Trying to abort 1 pending ops.
I20250627 13:53:13.184494 3789 catalog_manager.cc:5582] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1 reported cstate change: config changed from index 102 to 107, VOTER f46fd51df5864af9b8493acd69db0747 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } }
I20250627 13:53:13.188810 6095 pending_rounds.cc:77] T 37f6cffc2bc5456e94999b60340dfeb6 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Aborting op as it isn't in flight: id { term: 1 index: 109 } timestamp: 7172228681286688768 op_type: WRITE_OP write_request { tablet_id: "37f6cffc2bc5456e94999b60340dfeb6" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\355\336.PC\236K\317\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""e73daae5833daaae" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7172228681239326720 authz_token { token_data: "\010\356\306\372\302\006\"3\n\005slave\022*\n 41b84e234b64410a9ea4c057e1c398cd\020\001\030\001 \001(\001" signature: "<redacted>""F[\203\231;04RC\202\025\000\324\221\376\241\244\340}h\"\331\246o\017^\216!\\`#\351\240\317\262\200\306\206\010@\023\362\224\374\036\300\032\177>\351\323I\333\"4\t\000\010\3632|k\212\323" signing_key_seq_num: 0 } } request_id { client_id: "4574185a2c1847d39faa4f620b283870" seq_no: 299 first_incomplete_seq_no: 299 attempt_no: 0 }
I20250627 13:53:13.189445 6095 raft_consensus.cc:2270] T 37f6cffc2bc5456e94999b60340dfeb6 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:53:13.183203 6066 consensus_queue.cc:237] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 1.120, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } }
I20250627 13:53:13.195573 3912 tablet_service.cc:1515] Processing DeleteTablet for tablet 1cd6f75e462546c6bb05ef14af3498fa with delete_type TABLET_DATA_TOMBSTONED (TS f46fd51df5864af9b8493acd69db0747 not found in new config with opid_index 107) from {username='slave'} at 127.0.0.1:41436
I20250627 13:53:13.196091 5633 raft_consensus.cc:1273] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154 [term 1 LEARNER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 120. Preceding OpId from leader: term: 1 index: 121. (index mismatch)
I20250627 13:53:13.197238 5989 consensus_queue.cc:1035] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20250627 13:53:13.199469 3779 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 1cd6f75e462546c6bb05ef14af3498fa with cas_config_opid_index 102: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:53:13.202222 5633 raft_consensus.cc:2953] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154 [term 1 FOLLOWER]: Committing config change with OpId 1.121: config changed from index 74 to 121, 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } } }
I20250627 13:53:13.201404 5989 raft_consensus.cc:2953] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.121: config changed from index 74 to 121, 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } } }
I20250627 13:53:13.208112 3788 catalog_manager.cc:5582] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154 reported cstate change: config changed from index 74 to 121, 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } } }
I20250627 13:53:13.212080 6095 ts_tablet_manager.cc:1905] T 37f6cffc2bc5456e94999b60340dfeb6 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:53:13.226542 3930 raft_consensus.cc:1273] T be42f2b9b1b747adb641149f70da608a P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 120. Preceding OpId from leader: term: 1 index: 121. (index mismatch)
I20250627 13:53:13.233582 5985 raft_consensus.cc:2953] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab [term 1 LEADER]: Committing config change with OpId 1.122: config changed from index 121 to 122, VOTER f46fd51df5864af9b8493acd69db0747 (127.0.103.65) evicted. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } } }
I20250627 13:53:13.238174 3777 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 76518bc59dfe443abd8e97539d1bcb8a with cas_config_opid_index 121: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:53:13.240609 5633 raft_consensus.cc:2953] T 76518bc59dfe443abd8e97539d1bcb8a P 6503b59977444aaebe5ee70a0bc99154 [term 1 FOLLOWER]: Committing config change with OpId 1.122: config changed from index 121 to 122, VOTER f46fd51df5864af9b8493acd69db0747 (127.0.103.65) evicted. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } } }
I20250627 13:53:13.244240 4194 consensus_queue.cc:237] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 1.122, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } }
I20250627 13:53:13.240007 3789 catalog_manager.cc:5582] T 76518bc59dfe443abd8e97539d1bcb8a P ad800aa8621b4fc5938692202990b5ab reported cstate change: config changed from index 121 to 122, VOTER f46fd51df5864af9b8493acd69db0747 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "ad800aa8621b4fc5938692202990b5ab" committed_config { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:53:13.250174 5634 raft_consensus.cc:1273] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154 [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 122. Preceding OpId from leader: term: 1 index: 123. (index mismatch)
I20250627 13:53:13.253499 6060 consensus_queue.cc:1035] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20250627 13:53:13.249934 3912 tablet_service.cc:1515] Processing DeleteTablet for tablet 76518bc59dfe443abd8e97539d1bcb8a with delete_type TABLET_DATA_TOMBSTONED (TS f46fd51df5864af9b8493acd69db0747 not found in new config with opid_index 122) from {username='slave'} at 127.0.0.1:41436
I20250627 13:53:13.258653 6066 raft_consensus.cc:2953] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.123: config changed from index 121 to 123, VOTER f46fd51df5864af9b8493acd69db0747 (127.0.103.65) evicted. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } } }
I20250627 13:53:13.259621 5634 raft_consensus.cc:2953] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154 [term 1 FOLLOWER]: Committing config change with OpId 1.123: config changed from index 121 to 123, VOTER f46fd51df5864af9b8493acd69db0747 (127.0.103.65) evicted. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } } }
I20250627 13:53:13.266239 3779 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet be42f2b9b1b747adb641149f70da608a with cas_config_opid_index 121: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:53:13.267915 3789 catalog_manager.cc:5582] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 reported cstate change: config changed from index 121 to 123, VOTER f46fd51df5864af9b8493acd69db0747 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:53:13.323736 6097 ts_tablet_manager.cc:1918] T e79731c0edcc4c11a573cf46f0b2c06a P f46fd51df5864af9b8493acd69db0747: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.117
I20250627 13:53:13.368471 6095 ts_tablet_manager.cc:1918] T 37f6cffc2bc5456e94999b60340dfeb6 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.109
I20250627 13:53:13.368705 6095 log.cc:1199] T 37f6cffc2bc5456e94999b60340dfeb6 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal/wals/37f6cffc2bc5456e94999b60340dfeb6
I20250627 13:53:13.369671 3779 catalog_manager.cc:4928] TS fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67:44627): tablet 37f6cffc2bc5456e94999b60340dfeb6 (table rebalance_test_table_2 [id=41b84e234b64410a9ea4c057e1c398cd]) successfully deleted
I20250627 13:53:13.431908 6107 raft_consensus.cc:1062] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154: attempting to promote NON_VOTER 36100389495d4971b10cf590bc82c42c to VOTER
I20250627 13:53:13.432976 6107 consensus_queue.cc:237] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } }
I20250627 13:53:13.444687 5633 raft_consensus.cc:1215] T 978e20547d644d3988d6da29235903aa P 6503b59977444aaebe5ee70a0bc99154 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.82->[1.83-1.99] Dedup: 1.90->[1.91-1.99]
I20250627 13:53:13.458118 5240 raft_consensus.cc:1273] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 36100389495d4971b10cf590bc82c42c [term 2 LEARNER]: Refusing update from remote peer 6503b59977444aaebe5ee70a0bc99154: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250627 13:53:13.459142 6106 consensus_queue.cc:1035] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:53:13.332002 3912 tablet_service.cc:1515] Processing DeleteTablet for tablet be42f2b9b1b747adb641149f70da608a with delete_type TABLET_DATA_TOMBSTONED (TS f46fd51df5864af9b8493acd69db0747 not found in new config with opid_index 123) from {username='slave'} at 127.0.0.1:41436
I20250627 13:53:13.460808 3930 raft_consensus.cc:1273] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [term 2 FOLLOWER]: Refusing update from remote peer 6503b59977444aaebe5ee70a0bc99154: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250627 13:53:13.478535 6097 log.cc:1199] T e79731c0edcc4c11a573cf46f0b2c06a P f46fd51df5864af9b8493acd69db0747: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/wals/e79731c0edcc4c11a573cf46f0b2c06a
I20250627 13:53:13.478585 6103 consensus_queue.cc:1035] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:53:13.491595 6103 raft_consensus.cc:2953] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 36100389495d4971b10cf590bc82c42c (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } }
I20250627 13:53:13.495976 5239 raft_consensus.cc:2953] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 36100389495d4971b10cf590bc82c42c [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 36100389495d4971b10cf590bc82c42c (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } }
I20250627 13:53:13.501772 3788 catalog_manager.cc:5582] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 36100389495d4971b10cf590bc82c42c reported cstate change: config changed from index 3 to 4, 36100389495d4971b10cf590bc82c42c (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "6503b59977444aaebe5ee70a0bc99154" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } }
I20250627 13:53:13.535110 3776 catalog_manager.cc:4928] TS f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133): tablet e79731c0edcc4c11a573cf46f0b2c06a (table rebalance_test_table_1 [id=70cec1ac0b6b4a82a338f38f22668b39]) successfully deleted
I20250627 13:53:13.539750 6097 tablet_replica.cc:331] T 1cd6f75e462546c6bb05ef14af3498fa P f46fd51df5864af9b8493acd69db0747: stopping tablet replica
I20250627 13:53:13.539963 6097 raft_consensus.cc:2241] T 1cd6f75e462546c6bb05ef14af3498fa P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:53:13.540963 6097 raft_consensus.cc:2270] T 1cd6f75e462546c6bb05ef14af3498fa P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250627 13:53:13.548691 4128 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.71:45413, user_credentials={real_user=slave}} blocked reactor thread for 52216.7us
I20250627 13:53:13.562599 6097 ts_tablet_manager.cc:1905] T 1cd6f75e462546c6bb05ef14af3498fa P f46fd51df5864af9b8493acd69db0747: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:53:13.564556 6083 tablet_bootstrap.cc:492] T 4a118f20daaf45a2a1c539410dbf0d30 P 6503b59977444aaebe5ee70a0bc99154: Bootstrap replayed 1/1 log segments. Stats: ops{read=112 overwritten=0 applied=112 ignored=0} inserts{seen=797 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:53:13.565155 6083 tablet_bootstrap.cc:492] T 4a118f20daaf45a2a1c539410dbf0d30 P 6503b59977444aaebe5ee70a0bc99154: Bootstrap complete.
I20250627 13:53:13.565519 6083 ts_tablet_manager.cc:1397] T 4a118f20daaf45a2a1c539410dbf0d30 P 6503b59977444aaebe5ee70a0bc99154: Time spent bootstrapping tablet: real 0.563s user 0.200s sys 0.044s
I20250627 13:53:13.569393 6083 raft_consensus.cc:357] T 4a118f20daaf45a2a1c539410dbf0d30 P 6503b59977444aaebe5ee70a0bc99154 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: true } }
I20250627 13:53:13.569741 6083 raft_consensus.cc:738] T 4a118f20daaf45a2a1c539410dbf0d30 P 6503b59977444aaebe5ee70a0bc99154 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6503b59977444aaebe5ee70a0bc99154, State: Initialized, Role: LEARNER
I20250627 13:53:13.569967 6083 consensus_queue.cc:260] T 4a118f20daaf45a2a1c539410dbf0d30 P 6503b59977444aaebe5ee70a0bc99154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 112, Last appended: 1.112, Last appended by leader: 112, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: true } }
I20250627 13:53:13.570603 6083 ts_tablet_manager.cc:1428] T 4a118f20daaf45a2a1c539410dbf0d30 P 6503b59977444aaebe5ee70a0bc99154: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20250627 13:53:13.571672 4213 tablet_copy_service.cc:342] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Request end of tablet copy session 6503b59977444aaebe5ee70a0bc99154-4a118f20daaf45a2a1c539410dbf0d30 received from {username='slave'} at 127.0.103.71:57017
I20250627 13:53:13.571805 4213 tablet_copy_service.cc:434] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: ending tablet copy session 6503b59977444aaebe5ee70a0bc99154-4a118f20daaf45a2a1c539410dbf0d30 on tablet 4a118f20daaf45a2a1c539410dbf0d30 with peer 6503b59977444aaebe5ee70a0bc99154
I20250627 13:53:13.573496 6062 raft_consensus.cc:1062] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6: attempting to promote NON_VOTER 6503b59977444aaebe5ee70a0bc99154 to VOTER
I20250627 13:53:13.580456 6097 ts_tablet_manager.cc:1918] T 1cd6f75e462546c6bb05ef14af3498fa P f46fd51df5864af9b8493acd69db0747: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.106
I20250627 13:53:13.580677 6097 log.cc:1199] T 1cd6f75e462546c6bb05ef14af3498fa P f46fd51df5864af9b8493acd69db0747: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/wals/1cd6f75e462546c6bb05ef14af3498fa
I20250627 13:53:13.581452 6097 tablet_replica.cc:331] T 76518bc59dfe443abd8e97539d1bcb8a P f46fd51df5864af9b8493acd69db0747: stopping tablet replica
I20250627 13:53:13.581473 3776 catalog_manager.cc:4928] TS f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133): tablet 1cd6f75e462546c6bb05ef14af3498fa (table rebalance_test_table_3 [id=d219e402e8284b468ef8ca703004dd5f]) successfully deleted
I20250627 13:53:13.581776 6097 raft_consensus.cc:2241] T 76518bc59dfe443abd8e97539d1bcb8a P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:53:13.582697 6097 raft_consensus.cc:2270] T 76518bc59dfe443abd8e97539d1bcb8a P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:53:13.583961 6097 ts_tablet_manager.cc:1905] T 76518bc59dfe443abd8e97539d1bcb8a P f46fd51df5864af9b8493acd69db0747: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:53:13.597138 5972 tablet_bootstrap.cc:492] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f: Bootstrap replayed 1/1 log segments. Stats: ops{read=95 overwritten=0 applied=95 ignored=0} inserts{seen=726 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:53:13.597730 5972 tablet_bootstrap.cc:492] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f: Bootstrap complete.
I20250627 13:53:13.598074 5972 ts_tablet_manager.cc:1397] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f: Time spent bootstrapping tablet: real 1.467s user 0.152s sys 0.014s
I20250627 13:53:13.598934 5972 raft_consensus.cc:357] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: true } }
I20250627 13:53:13.599246 5972 raft_consensus.cc:738] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bf0be937c3634bcab62f1e58402a776f, State: Initialized, Role: LEARNER
I20250627 13:53:13.599481 5972 consensus_queue.cc:260] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 95, Last appended: 1.95, Last appended by leader: 95, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 88 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: true } }
I20250627 13:53:13.603761 6062 consensus_queue.cc:237] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 1.101, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } }
I20250627 13:53:13.608628 5972 ts_tablet_manager.cc:1428] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f: Time spent starting tablet: real 0.010s user 0.003s sys 0.004s
I20250627 13:53:13.621311 6097 ts_tablet_manager.cc:1918] T 76518bc59dfe443abd8e97539d1bcb8a P f46fd51df5864af9b8493acd69db0747: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.121
I20250627 13:53:13.621543 6097 log.cc:1199] T 76518bc59dfe443abd8e97539d1bcb8a P f46fd51df5864af9b8493acd69db0747: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/wals/76518bc59dfe443abd8e97539d1bcb8a
I20250627 13:53:13.621729 5634 raft_consensus.cc:1273] T 978e20547d644d3988d6da29235903aa P 6503b59977444aaebe5ee70a0bc99154 [term 1 LEARNER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 101. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20250627 13:53:13.622545 3776 catalog_manager.cc:4928] TS f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133): tablet 76518bc59dfe443abd8e97539d1bcb8a (table rebalance_test_table_3 [id=d219e402e8284b468ef8ca703004dd5f]) successfully deleted
I20250627 13:53:13.622891 6097 tablet_replica.cc:331] T be42f2b9b1b747adb641149f70da608a P f46fd51df5864af9b8493acd69db0747: stopping tablet replica
I20250627 13:53:13.622965 4064 raft_consensus.cc:1273] T 978e20547d644d3988d6da29235903aa P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 101. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20250627 13:53:13.623057 4213 tablet_copy_service.cc:342] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Request end of tablet copy session bf0be937c3634bcab62f1e58402a776f-9f374c0cbcd0493597c8f30e3e63d962 received from {username='slave'} at 127.0.103.70:49887
I20250627 13:53:13.623229 4213 tablet_copy_service.cc:434] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: ending tablet copy session bf0be937c3634bcab62f1e58402a776f-9f374c0cbcd0493597c8f30e3e63d962 on tablet 9f374c0cbcd0493597c8f30e3e63d962 with peer bf0be937c3634bcab62f1e58402a776f
I20250627 13:53:13.623330 6097 raft_consensus.cc:2241] T be42f2b9b1b747adb641149f70da608a P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:53:13.624102 5942 consensus_queue.cc:1035] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 100, Time since last communication: 0.000s
I20250627 13:53:13.624575 6097 raft_consensus.cc:2270] T be42f2b9b1b747adb641149f70da608a P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:53:13.625635 6097 ts_tablet_manager.cc:1905] T be42f2b9b1b747adb641149f70da608a P f46fd51df5864af9b8493acd69db0747: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:53:13.643280 3930 raft_consensus.cc:2953] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 36100389495d4971b10cf590bc82c42c (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } }
I20250627 13:53:13.653314 6097 ts_tablet_manager.cc:1918] T be42f2b9b1b747adb641149f70da608a P f46fd51df5864af9b8493acd69db0747: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.120
I20250627 13:53:13.663609 6060 consensus_queue.cc:1035] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 100, Time since last communication: 0.000s
I20250627 13:53:13.663715 6097 log.cc:1199] T be42f2b9b1b747adb641149f70da608a P f46fd51df5864af9b8493acd69db0747: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/wals/be42f2b9b1b747adb641149f70da608a
I20250627 13:53:13.665084 5634 raft_consensus.cc:1215] T 4a118f20daaf45a2a1c539410dbf0d30 P 6503b59977444aaebe5ee70a0bc99154 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.109->[1.110-1.121] Dedup: 1.112->[1.113-1.121]
I20250627 13:53:13.670126 5502 raft_consensus.cc:1215] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.87->[1.88-1.103] Dedup: 1.95->[1.96-1.103]
I20250627 13:53:13.681502 3776 catalog_manager.cc:4928] TS f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133): tablet be42f2b9b1b747adb641149f70da608a (table rebalance_test_table_1 [id=70cec1ac0b6b4a82a338f38f22668b39]) successfully deleted
I20250627 13:53:13.683898 6060 raft_consensus.cc:2953] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.102: config changed from index 83 to 102, 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } } }
I20250627 13:53:13.698359 5634 raft_consensus.cc:2953] T 978e20547d644d3988d6da29235903aa P 6503b59977444aaebe5ee70a0bc99154 [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index 83 to 102, 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } } }
I20250627 13:53:13.708461 5634 consensus_queue.cc:237] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } }
I20250627 13:53:13.710086 3784 catalog_manager.cc:5582] T 978e20547d644d3988d6da29235903aa P 6503b59977444aaebe5ee70a0bc99154 reported cstate change: config changed from index 83 to 102, 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } } }
I20250627 13:53:13.713939 5241 raft_consensus.cc:1273] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 36100389495d4971b10cf590bc82c42c [term 2 FOLLOWER]: Refusing update from remote peer 6503b59977444aaebe5ee70a0bc99154: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250627 13:53:13.714669 6107 consensus_queue.cc:1035] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:53:13.717149 6103 raft_consensus.cc:2953] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 6503b59977444aaebe5ee70a0bc99154 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER f46fd51df5864af9b8493acd69db0747 (127.0.103.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } }
I20250627 13:53:13.717689 5239 raft_consensus.cc:2953] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 36100389495d4971b10cf590bc82c42c [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER f46fd51df5864af9b8493acd69db0747 (127.0.103.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } }
I20250627 13:53:13.724499 3784 catalog_manager.cc:5582] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P 36100389495d4971b10cf590bc82c42c reported cstate change: config changed from index 4 to 5, VOTER f46fd51df5864af9b8493acd69db0747 (127.0.103.65) evicted. New cstate: current_term: 2 leader_uuid: "6503b59977444aaebe5ee70a0bc99154" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } }
I20250627 13:53:13.728677 3912 tablet_service.cc:1515] Processing DeleteTablet for tablet 0fdcc98da13c4cf998b1fd1b3e3f9fe5 with delete_type TABLET_DATA_TOMBSTONED (TS f46fd51df5864af9b8493acd69db0747 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41436
I20250627 13:53:13.729045 6097 tablet_replica.cc:331] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747: stopping tablet replica
I20250627 13:53:13.729223 6097 raft_consensus.cc:2241] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:53:13.729429 6097 raft_consensus.cc:2270] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:53:13.730522 6097 ts_tablet_manager.cc:1905] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:53:13.737478 6097 ts_tablet_manager.cc:1918] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250627 13:53:13.737699 6097 log.cc:1199] T 0fdcc98da13c4cf998b1fd1b3e3f9fe5 P f46fd51df5864af9b8493acd69db0747: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/wals/0fdcc98da13c4cf998b1fd1b3e3f9fe5
I20250627 13:53:13.738521 3776 catalog_manager.cc:4928] TS f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133): tablet 0fdcc98da13c4cf998b1fd1b3e3f9fe5 (table TestTable [id=d060a07fcd5b4f85b0fcfc50a67aee2c]) successfully deleted
I20250627 13:53:13.743662 3777 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 0fdcc98da13c4cf998b1fd1b3e3f9fe5 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:53:13.750857 4064 raft_consensus.cc:2953] T 978e20547d644d3988d6da29235903aa P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index 83 to 102, 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } } }
I20250627 13:53:13.769191 5942 raft_consensus.cc:1062] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: attempting to promote NON_VOTER 6503b59977444aaebe5ee70a0bc99154 to VOTER
W20250627 13:53:13.769780 6062 consensus_queue.cc:1162] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ad800aa8621b4fc5938692202990b5ab" responder_term: 1 status { last_received { term: 1 index: 122 } last_committed_idx: 121 last_received_current_leader { term: 1 index: 122 } }
I20250627 13:53:13.770085 5942 consensus_queue.cc:237] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 1.122, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } }
I20250627 13:53:13.776463 4194 consensus_queue.cc:237] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 1.103, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } }
I20250627 13:53:13.777287 5634 raft_consensus.cc:1273] T 4a118f20daaf45a2a1c539410dbf0d30 P 6503b59977444aaebe5ee70a0bc99154 [term 1 LEARNER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 122. Preceding OpId from leader: term: 1 index: 123. (index mismatch)
I20250627 13:53:13.779414 4062 raft_consensus.cc:1273] T 4a118f20daaf45a2a1c539410dbf0d30 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 122. Preceding OpId from leader: term: 1 index: 123. (index mismatch)
I20250627 13:53:13.779357 6062 consensus_queue.cc:1035] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 121, Time since last communication: 0.000s
I20250627 13:53:13.779856 6060 raft_consensus.cc:1062] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: attempting to promote NON_VOTER bf0be937c3634bcab62f1e58402a776f to VOTER
I20250627 13:53:13.780442 6060 consensus_queue.cc:237] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 1.104, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } }
I20250627 13:53:13.782582 5501 raft_consensus.cc:1273] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [term 1 LEARNER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch)
I20250627 13:53:13.783047 6062 consensus_queue.cc:1035] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 103, Time since last communication: 0.000s
I20250627 13:53:13.785537 6062 raft_consensus.cc:2953] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.105: config changed from index 88 to 105, bf0be937c3634bcab62f1e58402a776f (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } }
I20250627 13:53:13.786019 5501 raft_consensus.cc:2953] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index 88 to 105, bf0be937c3634bcab62f1e58402a776f (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } }
I20250627 13:53:13.791471 5634 raft_consensus.cc:1273] T 978e20547d644d3988d6da29235903aa P 6503b59977444aaebe5ee70a0bc99154 [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
I20250627 13:53:13.793419 6063 consensus_queue.cc:1035] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 121, Time since last communication: 0.000s
I20250627 13:53:13.793715 4062 raft_consensus.cc:1273] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 105. (index mismatch)
I20250627 13:53:13.794178 6063 consensus_queue.cc:1035] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 102, Time since last communication: 0.000s
I20250627 13:53:13.796411 6060 consensus_queue.cc:1035] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 102, Time since last communication: 0.000s
I20250627 13:53:13.797195 6066 raft_consensus.cc:2953] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.123: config changed from index 110 to 123, 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } } }
I20250627 13:53:13.799819 4062 raft_consensus.cc:2953] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index 88 to 105, bf0be937c3634bcab62f1e58402a776f (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } }
I20250627 13:53:13.807096 3784 catalog_manager.cc:5582] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f reported cstate change: config changed from index 88 to 105, bf0be937c3634bcab62f1e58402a776f (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } }
I20250627 13:53:13.808220 5942 raft_consensus.cc:2953] T 978e20547d644d3988d6da29235903aa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.104: config changed from index 102 to 104, VOTER ad800aa8621b4fc5938692202990b5ab (127.0.103.66) evicted. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } } }
I20250627 13:53:13.810882 5632 raft_consensus.cc:2953] T 978e20547d644d3988d6da29235903aa P 6503b59977444aaebe5ee70a0bc99154 [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index 102 to 104, VOTER ad800aa8621b4fc5938692202990b5ab (127.0.103.66) evicted. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } } }
I20250627 13:53:13.812922 3779 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 978e20547d644d3988d6da29235903aa with cas_config_opid_index 102: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:53:13.817601 4064 raft_consensus.cc:2953] T 4a118f20daaf45a2a1c539410dbf0d30 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Committing config change with OpId 1.123: config changed from index 110 to 123, 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } } }
I20250627 13:53:13.822412 5634 raft_consensus.cc:2953] T 4a118f20daaf45a2a1c539410dbf0d30 P 6503b59977444aaebe5ee70a0bc99154 [term 1 FOLLOWER]: Committing config change with OpId 1.123: config changed from index 110 to 123, 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } } }
I20250627 13:53:13.823565 3788 catalog_manager.cc:5582] T 4a118f20daaf45a2a1c539410dbf0d30 P ad800aa8621b4fc5938692202990b5ab reported cstate change: config changed from index 110 to 123, 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } } }
I20250627 13:53:13.842661 3788 catalog_manager.cc:5582] T 978e20547d644d3988d6da29235903aa P 6503b59977444aaebe5ee70a0bc99154 reported cstate change: config changed from index 102 to 104, VOTER ad800aa8621b4fc5938692202990b5ab (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } } }
I20250627 13:53:13.846166 4042 tablet_service.cc:1515] Processing DeleteTablet for tablet 978e20547d644d3988d6da29235903aa with delete_type TABLET_DATA_TOMBSTONED (TS ad800aa8621b4fc5938692202990b5ab not found in new config with opid_index 104) from {username='slave'} at 127.0.0.1:33136
I20250627 13:53:13.846938 4194 consensus_queue.cc:237] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 1.124, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } }
I20250627 13:53:13.850509 6142 tablet_replica.cc:331] T 978e20547d644d3988d6da29235903aa P ad800aa8621b4fc5938692202990b5ab: stopping tablet replica
I20250627 13:53:13.850785 6142 raft_consensus.cc:2241] T 978e20547d644d3988d6da29235903aa P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:53:13.851516 6142 raft_consensus.cc:2270] T 978e20547d644d3988d6da29235903aa P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:53:13.852738 6142 ts_tablet_manager.cc:1905] T 978e20547d644d3988d6da29235903aa P ad800aa8621b4fc5938692202990b5ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:53:13.853153 5634 raft_consensus.cc:1273] T 4a118f20daaf45a2a1c539410dbf0d30 P 6503b59977444aaebe5ee70a0bc99154 [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 124. Preceding OpId from leader: term: 1 index: 125. (index mismatch)
I20250627 13:53:13.854240 5942 consensus_queue.cc:1035] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 123, Time since last communication: 0.000s
I20250627 13:53:13.859119 5634 raft_consensus.cc:2953] T 4a118f20daaf45a2a1c539410dbf0d30 P 6503b59977444aaebe5ee70a0bc99154 [term 1 FOLLOWER]: Committing config change with OpId 1.125: config changed from index 123 to 125, VOTER ad800aa8621b4fc5938692202990b5ab (127.0.103.66) evicted. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } } }
I20250627 13:53:13.863756 6062 raft_consensus.cc:2953] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.125: config changed from index 123 to 125, VOTER ad800aa8621b4fc5938692202990b5ab (127.0.103.66) evicted. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } } }
I20250627 13:53:13.868363 3779 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 4a118f20daaf45a2a1c539410dbf0d30 with cas_config_opid_index 123: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:53:13.870002 3788 catalog_manager.cc:5582] T 4a118f20daaf45a2a1c539410dbf0d30 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 reported cstate change: config changed from index 123 to 125, VOTER ad800aa8621b4fc5938692202990b5ab (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:53:13.876581 6142 ts_tablet_manager.cc:1918] T 978e20547d644d3988d6da29235903aa P ad800aa8621b4fc5938692202990b5ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.102
I20250627 13:53:13.876807 6142 log.cc:1199] T 978e20547d644d3988d6da29235903aa P ad800aa8621b4fc5938692202990b5ab: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/wals/978e20547d644d3988d6da29235903aa
I20250627 13:53:13.877779 3777 catalog_manager.cc:4928] TS ad800aa8621b4fc5938692202990b5ab (127.0.103.66:43273): tablet 978e20547d644d3988d6da29235903aa (table rebalance_test_table_4 [id=b5d412773a2343d1919352385a3629db]) successfully deleted
I20250627 13:53:13.878978 4042 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a118f20daaf45a2a1c539410dbf0d30 with delete_type TABLET_DATA_TOMBSTONED (TS ad800aa8621b4fc5938692202990b5ab not found in new config with opid_index 125) from {username='slave'} at 127.0.0.1:33136
I20250627 13:53:13.879309 6142 tablet_replica.cc:331] T 4a118f20daaf45a2a1c539410dbf0d30 P ad800aa8621b4fc5938692202990b5ab: stopping tablet replica
I20250627 13:53:13.879534 6142 raft_consensus.cc:2241] T 4a118f20daaf45a2a1c539410dbf0d30 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:53:13.879909 6142 raft_consensus.cc:2270] T 4a118f20daaf45a2a1c539410dbf0d30 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:53:13.883949 6142 ts_tablet_manager.cc:1905] T 4a118f20daaf45a2a1c539410dbf0d30 P ad800aa8621b4fc5938692202990b5ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:53:13.896291 6142 ts_tablet_manager.cc:1918] T 4a118f20daaf45a2a1c539410dbf0d30 P ad800aa8621b4fc5938692202990b5ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.124
I20250627 13:53:13.896530 6142 log.cc:1199] T 4a118f20daaf45a2a1c539410dbf0d30 P ad800aa8621b4fc5938692202990b5ab: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/wals/4a118f20daaf45a2a1c539410dbf0d30
I20250627 13:53:13.898209 3777 catalog_manager.cc:4928] TS ad800aa8621b4fc5938692202990b5ab (127.0.103.66:43273): tablet 4a118f20daaf45a2a1c539410dbf0d30 (table rebalance_test_table_2 [id=41b84e234b64410a9ea4c057e1c398cd]) successfully deleted
I20250627 13:53:14.171733 4194 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "9f374c0cbcd0493597c8f30e3e63d962"
dest_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53370
I20250627 13:53:14.172012 4194 raft_consensus.cc:604] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Received request to transfer leadership
I20250627 13:53:14.173593 6063 raft_consensus.cc:991] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: : Instructing follower bf0be937c3634bcab62f1e58402a776f to start an election
I20250627 13:53:14.173780 6063 raft_consensus.cc:1079] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Signalling peer bf0be937c3634bcab62f1e58402a776f to start an election
I20250627 13:53:14.177217 5499 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "9f374c0cbcd0493597c8f30e3e63d962"
dest_uuid: "bf0be937c3634bcab62f1e58402a776f"
from {username='slave'} at 127.0.103.67:59171
I20250627 13:53:14.177758 5499 raft_consensus.cc:491] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:53:14.178102 5499 raft_consensus.cc:3058] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:53:14.181921 5499 raft_consensus.cc:513] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } }
I20250627 13:53:14.185235 5499 leader_election.cc:290] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [CANDIDATE]: Term 2 election: Requested vote from peers ad800aa8621b4fc5938692202990b5ab (127.0.103.66:43273), fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67:44627)
I20250627 13:53:14.185544 4194 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f374c0cbcd0493597c8f30e3e63d962" candidate_uuid: "bf0be937c3634bcab62f1e58402a776f" candidate_term: 2 candidate_status { last_received { term: 1 index: 112 } } ignore_live_leader: true dest_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6"
I20250627 13:53:14.185791 4194 raft_consensus.cc:3053] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Stepping down as leader of term 1
I20250627 13:53:14.185899 4194 raft_consensus.cc:738] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb586b8cb7ab44e1a7ec5016dbd2c0a6, State: Running, Role: LEADER
I20250627 13:53:14.186224 4194 consensus_queue.cc:260] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 1.112, Last appended by leader: 112, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } }
I20250627 13:53:14.186579 4194 raft_consensus.cc:3058] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:53:14.194151 4194 raft_consensus.cc:2466] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf0be937c3634bcab62f1e58402a776f in term 2.
W20250627 13:53:14.196432 4154 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53144: Illegal state: replica fb586b8cb7ab44e1a7ec5016dbd2c0a6 is not leader of this config: current role FOLLOWER
I20250627 13:53:14.206037 5438 leader_election.cc:304] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf0be937c3634bcab62f1e58402a776f, fb586b8cb7ab44e1a7ec5016dbd2c0a6; no voters:
I20250627 13:53:14.206557 6129 raft_consensus.cc:2802] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [term 2 FOLLOWER]: Leader election won for term 2
W20250627 13:53:14.214953 4022 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:39896: Illegal state: replica ad800aa8621b4fc5938692202990b5ab is not leader of this config: current role FOLLOWER
W20250627 13:53:14.225019 4154 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53144: Illegal state: replica fb586b8cb7ab44e1a7ec5016dbd2c0a6 is not leader of this config: current role FOLLOWER
W20250627 13:53:14.233317 4022 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:39896: Illegal state: replica ad800aa8621b4fc5938692202990b5ab is not leader of this config: current role FOLLOWER
I20250627 13:53:14.268677 6129 raft_consensus.cc:695] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [term 2 LEADER]: Becoming Leader. State: Replica: bf0be937c3634bcab62f1e58402a776f, State: Running, Role: LEADER
I20250627 13:53:14.269381 6129 consensus_queue.cc:237] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 1.112, Last appended by leader: 112, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } }
I20250627 13:53:14.276106 3784 catalog_manager.cc:5582] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f reported cstate change: term changed from 1 to 2, leader changed from fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67) to bf0be937c3634bcab62f1e58402a776f (127.0.103.70). New cstate: current_term: 2 leader_uuid: "bf0be937c3634bcab62f1e58402a776f" committed_config { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:53:14.315552 4194 raft_consensus.cc:1273] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 2 FOLLOWER]: Refusing update from remote peer bf0be937c3634bcab62f1e58402a776f: Log matching property violated. Preceding OpId in replica: term: 1 index: 112. Preceding OpId from leader: term: 2 index: 114. (index mismatch)
I20250627 13:53:14.316705 6129 consensus_queue.cc:1035] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20250627 13:53:14.352807 4062 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f374c0cbcd0493597c8f30e3e63d962" candidate_uuid: "bf0be937c3634bcab62f1e58402a776f" candidate_term: 2 candidate_status { last_received { term: 1 index: 112 } } ignore_live_leader: true dest_uuid: "ad800aa8621b4fc5938692202990b5ab"
I20250627 13:53:14.353016 4062 raft_consensus.cc:3058] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:53:14.356295 4062 raft_consensus.cc:2466] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf0be937c3634bcab62f1e58402a776f in term 2.
I20250627 13:53:14.356789 4064 raft_consensus.cc:1273] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab [term 2 FOLLOWER]: Refusing update from remote peer bf0be937c3634bcab62f1e58402a776f: Log matching property violated. Preceding OpId in replica: term: 1 index: 112. Preceding OpId from leader: term: 2 index: 114. (index mismatch)
I20250627 13:53:14.360145 6170 consensus_queue.cc:1035] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20250627 13:53:14.430823 5499 consensus_queue.cc:237] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 2.118, Last appended by leader: 112, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } }
I20250627 13:53:14.432428 4062 raft_consensus.cc:1273] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab [term 2 FOLLOWER]: Refusing update from remote peer bf0be937c3634bcab62f1e58402a776f: Log matching property violated. Preceding OpId in replica: term: 2 index: 118. Preceding OpId from leader: term: 2 index: 119. (index mismatch)
I20250627 13:53:14.433178 6129 consensus_queue.cc:1035] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20250627 13:53:14.436514 6129 raft_consensus.cc:2953] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [term 2 LEADER]: Committing config change with OpId 2.119: config changed from index 105 to 119, VOTER fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67) evicted. New config: { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } }
I20250627 13:53:14.443758 4064 raft_consensus.cc:2953] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab [term 2 FOLLOWER]: Committing config change with OpId 2.119: config changed from index 105 to 119, VOTER fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67) evicted. New config: { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } }
I20250627 13:53:14.452651 3779 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9f374c0cbcd0493597c8f30e3e63d962 with cas_config_opid_index 105: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:53:14.456308 3784 catalog_manager.cc:5582] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab reported cstate change: config changed from index 105 to 119, VOTER fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67) evicted. New cstate: current_term: 2 leader_uuid: "bf0be937c3634bcab62f1e58402a776f" committed_config { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } }
I20250627 13:53:14.481494 4174 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f374c0cbcd0493597c8f30e3e63d962 with delete_type TABLET_DATA_TOMBSTONED (TS fb586b8cb7ab44e1a7ec5016dbd2c0a6 not found in new config with opid_index 119) from {username='slave'} at 127.0.0.1:32882
I20250627 13:53:14.513394 6188 tablet_replica.cc:331] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: stopping tablet replica
I20250627 13:53:14.513779 6188 raft_consensus.cc:2241] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:53:14.514057 6188 raft_consensus.cc:2270] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:53:14.515076 6188 ts_tablet_manager.cc:1905] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:53:14.533255 6188 ts_tablet_manager.cc:1918] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.118
I20250627 13:53:14.535787 6188 log.cc:1199] T 9f374c0cbcd0493597c8f30e3e63d962 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal/wals/9f374c0cbcd0493597c8f30e3e63d962
I20250627 13:53:14.537482 3779 catalog_manager.cc:4928] TS fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67:44627): tablet 9f374c0cbcd0493597c8f30e3e63d962 (table rebalance_test_table_0 [id=59a8f27f17c04730859e43c3fde21ee6]) successfully deleted
I20250627 13:53:14.659158 4194 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "3e2b11c6788e4ccdbdde60769fee1aae"
dest_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53408
I20250627 13:53:14.659394 4194 raft_consensus.cc:604] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Received request to transfer leadership
W20250627 13:53:14.676581 4152 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53160: Service unavailable: leader transfer in progress
W20250627 13:53:14.695806 4152 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53160: Service unavailable: leader transfer in progress
W20250627 13:53:14.707906 4153 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53160: Service unavailable: leader transfer in progress
W20250627 13:53:14.714941 4153 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53160: Service unavailable: leader transfer in progress
W20250627 13:53:14.744057 4153 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53160: Service unavailable: leader transfer in progress
W20250627 13:53:14.755978 4153 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53160: Service unavailable: leader transfer in progress
W20250627 13:53:14.765112 4153 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53160: Service unavailable: leader transfer in progress
W20250627 13:53:14.783435 4153 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53160: Service unavailable: leader transfer in progress
W20250627 13:53:14.812429 4153 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53160: Service unavailable: leader transfer in progress
W20250627 13:53:14.827149 4153 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53160: Service unavailable: leader transfer in progress
W20250627 13:53:14.841996 4153 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53160: Service unavailable: leader transfer in progress
W20250627 13:53:14.862787 4153 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53160: Service unavailable: leader transfer in progress
W20250627 13:53:14.880637 4153 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53160: Service unavailable: leader transfer in progress
W20250627 13:53:14.902297 4153 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53160: Service unavailable: leader transfer in progress
W20250627 13:53:14.922488 4153 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53160: Service unavailable: leader transfer in progress
W20250627 13:53:14.941251 4153 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53160: Service unavailable: leader transfer in progress
W20250627 13:53:14.966821 4153 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53160: Service unavailable: leader transfer in progress
W20250627 13:53:14.989878 4153 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53160: Service unavailable: leader transfer in progress
W20250627 13:53:15.013695 4153 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53160: Service unavailable: leader transfer in progress
W20250627 13:53:15.039669 4153 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53160: Service unavailable: leader transfer in progress
W20250627 13:53:15.063642 4154 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53160: Service unavailable: leader transfer in progress
W20250627 13:53:15.088481 4153 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53160: Service unavailable: leader transfer in progress
W20250627 13:53:15.115839 4153 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53160: Service unavailable: leader transfer in progress
W20250627 13:53:15.144511 4153 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53160: Service unavailable: leader transfer in progress
W20250627 13:53:15.176466 4154 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53160: Service unavailable: leader transfer in progress
I20250627 13:53:15.199404 4194 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "3e2b11c6788e4ccdbdde60769fee1aae"
dest_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53446
I20250627 13:53:15.199710 4194 raft_consensus.cc:604] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Received request to transfer leadership
I20250627 13:53:15.212049 6066 raft_consensus.cc:991] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6: : Instructing follower bf0be937c3634bcab62f1e58402a776f to start an election
I20250627 13:53:15.212234 6066 raft_consensus.cc:1079] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Signalling peer bf0be937c3634bcab62f1e58402a776f to start an election
I20250627 13:53:15.213788 5501 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "3e2b11c6788e4ccdbdde60769fee1aae"
dest_uuid: "bf0be937c3634bcab62f1e58402a776f"
from {username='slave'} at 127.0.103.67:59171
W20250627 13:53:15.213884 4153 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53160: Service unavailable: leader transfer in progress
I20250627 13:53:15.213994 5501 raft_consensus.cc:491] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:53:15.214666 5501 raft_consensus.cc:3058] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:53:15.232623 5501 raft_consensus.cc:513] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } }
I20250627 13:53:15.233959 4194 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e2b11c6788e4ccdbdde60769fee1aae" candidate_uuid: "bf0be937c3634bcab62f1e58402a776f" candidate_term: 2 candidate_status { last_received { term: 1 index: 149 } } ignore_live_leader: true dest_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6"
I20250627 13:53:15.234041 5501 leader_election.cc:290] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f [CANDIDATE]: Term 2 election: Requested vote from peers fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67:44627), f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133)
I20250627 13:53:15.234181 4194 raft_consensus.cc:3053] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Stepping down as leader of term 1
I20250627 13:53:15.234263 4194 raft_consensus.cc:738] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb586b8cb7ab44e1a7ec5016dbd2c0a6, State: Running, Role: LEADER
I20250627 13:53:15.234758 4194 consensus_queue.cc:260] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 149, Committed index: 149, Last appended: 1.149, Last appended by leader: 149, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } }
I20250627 13:53:15.235136 4194 raft_consensus.cc:3058] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:53:15.239535 3932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3e2b11c6788e4ccdbdde60769fee1aae" candidate_uuid: "bf0be937c3634bcab62f1e58402a776f" candidate_term: 2 candidate_status { last_received { term: 1 index: 149 } } ignore_live_leader: true dest_uuid: "f46fd51df5864af9b8493acd69db0747"
I20250627 13:53:15.239796 3932 raft_consensus.cc:3058] T 3e2b11c6788e4ccdbdde60769fee1aae P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:53:15.240921 4194 raft_consensus.cc:2466] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf0be937c3634bcab62f1e58402a776f in term 2.
I20250627 13:53:15.241638 5438 leader_election.cc:304] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf0be937c3634bcab62f1e58402a776f, fb586b8cb7ab44e1a7ec5016dbd2c0a6; no voters:
I20250627 13:53:15.242482 6129 raft_consensus.cc:2802] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:53:15.242622 6129 raft_consensus.cc:695] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f [term 2 LEADER]: Becoming Leader. State: Replica: bf0be937c3634bcab62f1e58402a776f, State: Running, Role: LEADER
I20250627 13:53:15.242888 6129 consensus_queue.cc:237] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 149, Committed index: 149, Last appended: 1.149, Last appended by leader: 149, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } }
I20250627 13:53:15.245613 3788 catalog_manager.cc:5582] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f reported cstate change: term changed from 1 to 2, leader changed from fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67) to bf0be937c3634bcab62f1e58402a776f (127.0.103.70). New cstate: current_term: 2 leader_uuid: "bf0be937c3634bcab62f1e58402a776f" committed_config { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:53:15.246059 3932 raft_consensus.cc:2466] T 3e2b11c6788e4ccdbdde60769fee1aae P f46fd51df5864af9b8493acd69db0747 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf0be937c3634bcab62f1e58402a776f in term 2.
W20250627 13:53:15.250806 4153 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53160: Illegal state: replica fb586b8cb7ab44e1a7ec5016dbd2c0a6 is not leader of this config: current role FOLLOWER
W20250627 13:53:15.330137 3892 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:41372: Illegal state: replica f46fd51df5864af9b8493acd69db0747 is not leader of this config: current role FOLLOWER
I20250627 13:53:15.424264 4194 raft_consensus.cc:1273] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 2 FOLLOWER]: Refusing update from remote peer bf0be937c3634bcab62f1e58402a776f: Log matching property violated. Preceding OpId in replica: term: 1 index: 149. Preceding OpId from leader: term: 2 index: 151. (index mismatch)
I20250627 13:53:15.425439 6129 consensus_queue.cc:1035] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 150, Last known committed idx: 149, Time since last communication: 0.000s
I20250627 13:53:15.426210 3932 raft_consensus.cc:1273] T 3e2b11c6788e4ccdbdde60769fee1aae P f46fd51df5864af9b8493acd69db0747 [term 2 FOLLOWER]: Refusing update from remote peer bf0be937c3634bcab62f1e58402a776f: Log matching property violated. Preceding OpId in replica: term: 1 index: 149. Preceding OpId from leader: term: 2 index: 151. (index mismatch)
I20250627 13:53:15.428752 6129 consensus_queue.cc:1035] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 150, Last known committed idx: 149, Time since last communication: 0.000s
I20250627 13:53:15.472374 5820 mvcc.cc:204] Tried to move back new op lower bound from 7172228691649200128 to 7172228690917445632. Current Snapshot: MvccSnapshot[applied={T|T < 7172228691649200128}]
I20250627 13:53:15.476661 5499 consensus_queue.cc:237] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 152, Committed index: 152, Last appended: 2.152, Last appended by leader: 149, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 153 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } }
I20250627 13:53:15.482915 3931 raft_consensus.cc:1273] T 3e2b11c6788e4ccdbdde60769fee1aae P f46fd51df5864af9b8493acd69db0747 [term 2 FOLLOWER]: Refusing update from remote peer bf0be937c3634bcab62f1e58402a776f: Log matching property violated. Preceding OpId in replica: term: 2 index: 152. Preceding OpId from leader: term: 2 index: 153. (index mismatch)
I20250627 13:53:15.488507 6194 consensus_queue.cc:1035] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 153, Last known committed idx: 152, Time since last communication: 0.000s
I20250627 13:53:15.493678 6129 raft_consensus.cc:2953] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f [term 2 LEADER]: Committing config change with OpId 2.153: config changed from index 115 to 153, VOTER fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67) evicted. New config: { opid_index: 153 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } }
I20250627 13:53:15.497823 3779 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 3e2b11c6788e4ccdbdde60769fee1aae with cas_config_opid_index 115: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:53:15.506063 3788 catalog_manager.cc:5582] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f reported cstate change: config changed from index 115 to 153, VOTER fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67) evicted. New cstate: current_term: 2 leader_uuid: "bf0be937c3634bcab62f1e58402a776f" committed_config { opid_index: 153 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:53:15.510844 4174 tablet_service.cc:1515] Processing DeleteTablet for tablet 3e2b11c6788e4ccdbdde60769fee1aae with delete_type TABLET_DATA_TOMBSTONED (TS fb586b8cb7ab44e1a7ec5016dbd2c0a6 not found in new config with opid_index 153) from {username='slave'} at 127.0.0.1:32882
I20250627 13:53:15.517833 3932 raft_consensus.cc:2953] T 3e2b11c6788e4ccdbdde60769fee1aae P f46fd51df5864af9b8493acd69db0747 [term 2 FOLLOWER]: Committing config change with OpId 2.153: config changed from index 115 to 153, VOTER fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67) evicted. New config: { opid_index: 153 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } }
I20250627 13:53:15.525256 6271 tablet_replica.cc:331] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6: stopping tablet replica
I20250627 13:53:15.525534 6271 raft_consensus.cc:2241] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:53:15.525856 6271 raft_consensus.cc:2270] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:53:15.537395 6271 ts_tablet_manager.cc:1905] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:53:15.554216 6271 ts_tablet_manager.cc:1918] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.152
I20250627 13:53:15.562662 6271 log.cc:1199] T 3e2b11c6788e4ccdbdde60769fee1aae P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal/wals/3e2b11c6788e4ccdbdde60769fee1aae
I20250627 13:53:15.565225 3779 catalog_manager.cc:4928] TS fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67:44627): tablet 3e2b11c6788e4ccdbdde60769fee1aae (table rebalance_test_table_3 [id=d219e402e8284b468ef8ca703004dd5f]) successfully deleted
I20250627 13:53:16.252740 5221 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:53:16.299062 5614 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:16.332823 4042 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:53:16.335752 5482 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
I20250627 13:53:16.351354 5352 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:53:16.436666 3912 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
I20250627 13:53:16.547132 4174 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:53:16.877044 4194 consensus_queue.cc:237] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 166, Committed index: 166, Last appended: 1.166, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: true } }
I20250627 13:53:16.879942 5239 raft_consensus.cc:1273] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 166. Preceding OpId from leader: term: 1 index: 167. (index mismatch)
I20250627 13:53:16.881836 6060 consensus_queue.cc:1035] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 167, Last known committed idx: 166, Time since last communication: 0.000s
I20250627 13:53:16.885007 5239 raft_consensus.cc:2953] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c [term 1 FOLLOWER]: Committing config change with OpId 1.167: config changed from index 106 to 167, NON_VOTER 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69) added. New config: { opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: true } } }
W20250627 13:53:16.885574 4128 consensus_peers.cc:487] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 -> Peer 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69:42665): Couldn't send request to peer 911714a383074e09a9b2d127ab46d2c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8fe13ad746124b1d82cfb8ce691828a4. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:53:16.884373 6060 raft_consensus.cc:2953] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.167: config changed from index 106 to 167, NON_VOTER 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69) added. New config: { opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: true } } }
I20250627 13:53:16.926146 3788 catalog_manager.cc:5582] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 reported cstate change: config changed from index 106 to 167, NON_VOTER 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69) added. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:17.004992 5499 consensus_queue.cc:237] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 179, Committed index: 179, Last appended: 2.179, Last appended by leader: 112, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: true } }
I20250627 13:53:17.014034 4064 raft_consensus.cc:1273] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab [term 2 FOLLOWER]: Refusing update from remote peer bf0be937c3634bcab62f1e58402a776f: Log matching property violated. Preceding OpId in replica: term: 2 index: 179. Preceding OpId from leader: term: 2 index: 181. (index mismatch)
I20250627 13:53:17.014597 6269 consensus_queue.cc:1035] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 180, Last known committed idx: 179, Time since last communication: 0.000s
I20250627 13:53:17.017587 6269 raft_consensus.cc:2953] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [term 2 LEADER]: Committing config change with OpId 2.180: config changed from index 119 to 180, NON_VOTER 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69) added. New config: { opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: true } } }
I20250627 13:53:17.019909 4064 raft_consensus.cc:2953] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab [term 2 FOLLOWER]: Committing config change with OpId 2.180: config changed from index 119 to 180, NON_VOTER 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69) added. New config: { opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: true } } }
I20250627 13:53:17.025381 3788 catalog_manager.cc:5582] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f reported cstate change: config changed from index 119 to 180, NON_VOTER 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69) added. New cstate: current_term: 2 leader_uuid: "bf0be937c3634bcab62f1e58402a776f" committed_config { opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:17.082922 4194 consensus_queue.cc:237] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 178, Committed index: 178, Last appended: 1.178, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: true } }
W20250627 13:53:17.089524 4130 consensus_peers.cc:487] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 -> Peer bf0be937c3634bcab62f1e58402a776f (127.0.103.70:37215): Couldn't send request to peer bf0be937c3634bcab62f1e58402a776f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e79731c0edcc4c11a573cf46f0b2c06a. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:53:17.092193 5241 raft_consensus.cc:1273] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 178. Preceding OpId from leader: term: 1 index: 179. (index mismatch)
I20250627 13:53:17.092862 6060 consensus_queue.cc:1035] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 179, Last known committed idx: 178, Time since last communication: 0.000s
I20250627 13:53:17.098220 6341 raft_consensus.cc:2953] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.179: config changed from index 118 to 179, NON_VOTER bf0be937c3634bcab62f1e58402a776f (127.0.103.70) added. New config: { opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: true } } }
I20250627 13:53:17.107066 3788 catalog_manager.cc:5582] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 reported cstate change: config changed from index 118 to 179, NON_VOTER bf0be937c3634bcab62f1e58402a776f (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250627 13:53:17.114346 5436 consensus_peers.cc:487] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f -> Peer 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69:42665): Couldn't send request to peer 911714a383074e09a9b2d127ab46d2c1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9f374c0cbcd0493597c8f30e3e63d962. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:53:17.110909 5241 raft_consensus.cc:2953] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c [term 1 FOLLOWER]: Committing config change with OpId 1.179: config changed from index 118 to 179, NON_VOTER bf0be937c3634bcab62f1e58402a776f (127.0.103.70) added. New config: { opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: true } } }
I20250627 13:53:17.433599 6378 ts_tablet_manager.cc:927] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1: Initiating tablet copy from peer fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67:44627)
I20250627 13:53:17.488667 6378 tablet_copy_client.cc:323] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:44627
I20250627 13:53:17.492247 4213 tablet_copy_service.cc:140] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Received BeginTabletCopySession request for tablet 8fe13ad746124b1d82cfb8ce691828a4 from peer 911714a383074e09a9b2d127ab46d2c1 ({username='slave'} at 127.0.103.69:36137)
I20250627 13:53:17.492437 4213 tablet_copy_service.cc:161] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Beginning new tablet copy session on tablet 8fe13ad746124b1d82cfb8ce691828a4 from peer 911714a383074e09a9b2d127ab46d2c1 at {username='slave'} at 127.0.103.69:36137: session id = 911714a383074e09a9b2d127ab46d2c1-8fe13ad746124b1d82cfb8ce691828a4
I20250627 13:53:17.495287 4213 tablet_copy_source_session.cc:215] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:53:17.498170 6378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fe13ad746124b1d82cfb8ce691828a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:17.516966 6378 tablet_copy_client.cc:806] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1: tablet copy: Starting download of 0 data blocks...
I20250627 13:53:17.519003 6378 tablet_copy_client.cc:670] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1: tablet copy: Starting download of 1 WAL segments...
I20250627 13:53:17.557047 6389 ts_tablet_manager.cc:927] T e79731c0edcc4c11a573cf46f0b2c06a P bf0be937c3634bcab62f1e58402a776f: Initiating tablet copy from peer fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67:44627)
I20250627 13:53:17.574352 6378 tablet_copy_client.cc:538] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:53:17.589787 6378 tablet_bootstrap.cc:492] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1: Bootstrap starting.
I20250627 13:53:17.601387 6390 ts_tablet_manager.cc:927] T 9f374c0cbcd0493597c8f30e3e63d962 P 911714a383074e09a9b2d127ab46d2c1: Initiating tablet copy from peer bf0be937c3634bcab62f1e58402a776f (127.0.103.70:37215)
I20250627 13:53:17.628898 6390 tablet_copy_client.cc:323] T 9f374c0cbcd0493597c8f30e3e63d962 P 911714a383074e09a9b2d127ab46d2c1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.70:37215
I20250627 13:53:17.661900 5522 tablet_copy_service.cc:140] P bf0be937c3634bcab62f1e58402a776f: Received BeginTabletCopySession request for tablet 9f374c0cbcd0493597c8f30e3e63d962 from peer 911714a383074e09a9b2d127ab46d2c1 ({username='slave'} at 127.0.103.69:53001)
I20250627 13:53:17.662139 5522 tablet_copy_service.cc:161] P bf0be937c3634bcab62f1e58402a776f: Beginning new tablet copy session on tablet 9f374c0cbcd0493597c8f30e3e63d962 from peer 911714a383074e09a9b2d127ab46d2c1 at {username='slave'} at 127.0.103.69:53001: session id = 911714a383074e09a9b2d127ab46d2c1-9f374c0cbcd0493597c8f30e3e63d962
I20250627 13:53:17.676548 5522 tablet_copy_source_session.cc:215] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:53:17.694841 6390 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f374c0cbcd0493597c8f30e3e63d962. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:17.711290 6389 tablet_copy_client.cc:323] T e79731c0edcc4c11a573cf46f0b2c06a P bf0be937c3634bcab62f1e58402a776f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:44627
I20250627 13:53:17.712428 4213 tablet_copy_service.cc:140] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Received BeginTabletCopySession request for tablet e79731c0edcc4c11a573cf46f0b2c06a from peer bf0be937c3634bcab62f1e58402a776f ({username='slave'} at 127.0.103.70:49887)
I20250627 13:53:17.712617 4213 tablet_copy_service.cc:161] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Beginning new tablet copy session on tablet e79731c0edcc4c11a573cf46f0b2c06a from peer bf0be937c3634bcab62f1e58402a776f at {username='slave'} at 127.0.103.70:49887: session id = bf0be937c3634bcab62f1e58402a776f-e79731c0edcc4c11a573cf46f0b2c06a
I20250627 13:53:17.715978 4213 tablet_copy_source_session.cc:215] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:53:17.724017 6390 tablet_copy_client.cc:806] T 9f374c0cbcd0493597c8f30e3e63d962 P 911714a383074e09a9b2d127ab46d2c1: tablet copy: Starting download of 0 data blocks...
I20250627 13:53:17.746320 6390 tablet_copy_client.cc:670] T 9f374c0cbcd0493597c8f30e3e63d962 P 911714a383074e09a9b2d127ab46d2c1: tablet copy: Starting download of 1 WAL segments...
I20250627 13:53:17.746592 6389 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e79731c0edcc4c11a573cf46f0b2c06a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:17.749162 6390 tablet_copy_client.cc:538] T 9f374c0cbcd0493597c8f30e3e63d962 P 911714a383074e09a9b2d127ab46d2c1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:53:17.781442 6390 tablet_bootstrap.cc:492] T 9f374c0cbcd0493597c8f30e3e63d962 P 911714a383074e09a9b2d127ab46d2c1: Bootstrap starting.
I20250627 13:53:17.785354 6389 tablet_copy_client.cc:806] T e79731c0edcc4c11a573cf46f0b2c06a P bf0be937c3634bcab62f1e58402a776f: tablet copy: Starting download of 0 data blocks...
I20250627 13:53:17.785780 6389 tablet_copy_client.cc:670] T e79731c0edcc4c11a573cf46f0b2c06a P bf0be937c3634bcab62f1e58402a776f: tablet copy: Starting download of 1 WAL segments...
I20250627 13:53:17.790710 6389 tablet_copy_client.cc:538] T e79731c0edcc4c11a573cf46f0b2c06a P bf0be937c3634bcab62f1e58402a776f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:53:17.796092 6389 tablet_bootstrap.cc:492] T e79731c0edcc4c11a573cf46f0b2c06a P bf0be937c3634bcab62f1e58402a776f: Bootstrap starting.
I20250627 13:53:18.281605 6378 tablet_bootstrap.cc:492] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=173 overwritten=0 applied=173 ignored=0} inserts{seen=801 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:53:18.290932 6378 tablet_bootstrap.cc:492] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1: Bootstrap complete.
I20250627 13:53:18.291635 6378 ts_tablet_manager.cc:1397] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1: Time spent bootstrapping tablet: real 0.705s user 0.238s sys 0.025s
I20250627 13:53:18.293413 6378 raft_consensus.cc:357] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: true } }
I20250627 13:53:18.294139 6378 raft_consensus.cc:738] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 911714a383074e09a9b2d127ab46d2c1, State: Initialized, Role: LEARNER
I20250627 13:53:18.294596 6378 consensus_queue.cc:260] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 173, Last appended: 1.173, Last appended by leader: 173, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: true } }
I20250627 13:53:18.298789 6378 ts_tablet_manager.cc:1428] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20250627 13:53:18.305261 4213 tablet_copy_service.cc:342] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Request end of tablet copy session 911714a383074e09a9b2d127ab46d2c1-8fe13ad746124b1d82cfb8ce691828a4 received from {username='slave'} at 127.0.103.69:36137
I20250627 13:53:18.305435 4213 tablet_copy_service.cc:434] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: ending tablet copy session 911714a383074e09a9b2d127ab46d2c1-8fe13ad746124b1d82cfb8ce691828a4 on tablet 8fe13ad746124b1d82cfb8ce691828a4 with peer 911714a383074e09a9b2d127ab46d2c1
I20250627 13:53:18.472510 6390 tablet_bootstrap.cc:492] T 9f374c0cbcd0493597c8f30e3e63d962 P 911714a383074e09a9b2d127ab46d2c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=195 overwritten=0 applied=195 ignored=0} inserts{seen=822 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:53:18.473151 6390 tablet_bootstrap.cc:492] T 9f374c0cbcd0493597c8f30e3e63d962 P 911714a383074e09a9b2d127ab46d2c1: Bootstrap complete.
I20250627 13:53:18.473522 6390 ts_tablet_manager.cc:1397] T 9f374c0cbcd0493597c8f30e3e63d962 P 911714a383074e09a9b2d127ab46d2c1: Time spent bootstrapping tablet: real 0.693s user 0.325s sys 0.131s
I20250627 13:53:18.474447 6390 raft_consensus.cc:357] T 9f374c0cbcd0493597c8f30e3e63d962 P 911714a383074e09a9b2d127ab46d2c1 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: true } }
I20250627 13:53:18.474766 6390 raft_consensus.cc:738] T 9f374c0cbcd0493597c8f30e3e63d962 P 911714a383074e09a9b2d127ab46d2c1 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 911714a383074e09a9b2d127ab46d2c1, State: Initialized, Role: LEARNER
I20250627 13:53:18.475031 6390 consensus_queue.cc:260] T 9f374c0cbcd0493597c8f30e3e63d962 P 911714a383074e09a9b2d127ab46d2c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 195, Last appended: 2.195, Last appended by leader: 195, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: true } }
I20250627 13:53:18.475662 6390 ts_tablet_manager.cc:1428] T 9f374c0cbcd0493597c8f30e3e63d962 P 911714a383074e09a9b2d127ab46d2c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250627 13:53:18.478610 6389 tablet_bootstrap.cc:492] T e79731c0edcc4c11a573cf46f0b2c06a P bf0be937c3634bcab62f1e58402a776f: Bootstrap replayed 1/1 log segments. Stats: ops{read=189 overwritten=0 applied=188 ignored=0} inserts{seen=816 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250627 13:53:18.479120 6389 tablet_bootstrap.cc:492] T e79731c0edcc4c11a573cf46f0b2c06a P bf0be937c3634bcab62f1e58402a776f: Bootstrap complete.
I20250627 13:53:18.479387 6389 ts_tablet_manager.cc:1397] T e79731c0edcc4c11a573cf46f0b2c06a P bf0be937c3634bcab62f1e58402a776f: Time spent bootstrapping tablet: real 0.683s user 0.230s sys 0.040s
I20250627 13:53:18.480134 5522 tablet_copy_service.cc:342] P bf0be937c3634bcab62f1e58402a776f: Request end of tablet copy session 911714a383074e09a9b2d127ab46d2c1-9f374c0cbcd0493597c8f30e3e63d962 received from {username='slave'} at 127.0.103.69:53001
I20250627 13:53:18.480372 5522 tablet_copy_service.cc:434] P bf0be937c3634bcab62f1e58402a776f: ending tablet copy session 911714a383074e09a9b2d127ab46d2c1-9f374c0cbcd0493597c8f30e3e63d962 on tablet 9f374c0cbcd0493597c8f30e3e63d962 with peer 911714a383074e09a9b2d127ab46d2c1
I20250627 13:53:18.480274 6389 raft_consensus.cc:357] T e79731c0edcc4c11a573cf46f0b2c06a P bf0be937c3634bcab62f1e58402a776f [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: true } }
I20250627 13:53:18.481565 6389 raft_consensus.cc:738] T e79731c0edcc4c11a573cf46f0b2c06a P bf0be937c3634bcab62f1e58402a776f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bf0be937c3634bcab62f1e58402a776f, State: Initialized, Role: LEARNER
I20250627 13:53:18.481779 6389 consensus_queue.cc:260] T e79731c0edcc4c11a573cf46f0b2c06a P bf0be937c3634bcab62f1e58402a776f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 188, Last appended: 1.189, Last appended by leader: 189, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 179 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: true } }
I20250627 13:53:18.482378 6389 ts_tablet_manager.cc:1428] T e79731c0edcc4c11a573cf46f0b2c06a P bf0be937c3634bcab62f1e58402a776f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250627 13:53:18.486704 4213 tablet_copy_service.cc:342] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Request end of tablet copy session bf0be937c3634bcab62f1e58402a776f-e79731c0edcc4c11a573cf46f0b2c06a received from {username='slave'} at 127.0.103.70:49887
I20250627 13:53:18.486901 4213 tablet_copy_service.cc:434] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: ending tablet copy session bf0be937c3634bcab62f1e58402a776f-e79731c0edcc4c11a573cf46f0b2c06a on tablet e79731c0edcc4c11a573cf46f0b2c06a with peer bf0be937c3634bcab62f1e58402a776f
I20250627 13:53:18.606958 5372 raft_consensus.cc:1215] T 9f374c0cbcd0493597c8f30e3e63d962 P 911714a383074e09a9b2d127ab46d2c1 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.179->[2.180-2.205] Dedup: 2.195->[2.196-2.205]
I20250627 13:53:18.776046 5370 raft_consensus.cc:1215] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.166->[1.167-1.196] Dedup: 1.173->[1.174-1.196]
I20250627 13:53:18.837456 5499 raft_consensus.cc:1215] T e79731c0edcc4c11a573cf46f0b2c06a P bf0be937c3634bcab62f1e58402a776f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.178->[1.179-1.202] Dedup: 1.189->[1.190-1.202]
I20250627 13:53:18.848731 6194 raft_consensus.cc:1062] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f: attempting to promote NON_VOTER 911714a383074e09a9b2d127ab46d2c1 to VOTER
I20250627 13:53:18.849634 6194 consensus_queue.cc:237] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 212, Committed index: 212, Last appended: 2.212, Last appended by leader: 112, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } }
I20250627 13:53:18.885149 4064 raft_consensus.cc:1273] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab [term 2 FOLLOWER]: Refusing update from remote peer bf0be937c3634bcab62f1e58402a776f: Log matching property violated. Preceding OpId in replica: term: 2 index: 212. Preceding OpId from leader: term: 2 index: 213. (index mismatch)
I20250627 13:53:18.885775 6194 consensus_queue.cc:1035] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 212, Time since last communication: 0.000s
I20250627 13:53:18.885970 5370 raft_consensus.cc:1273] T 9f374c0cbcd0493597c8f30e3e63d962 P 911714a383074e09a9b2d127ab46d2c1 [term 2 LEARNER]: Refusing update from remote peer bf0be937c3634bcab62f1e58402a776f: Log matching property violated. Preceding OpId in replica: term: 2 index: 212. Preceding OpId from leader: term: 2 index: 213. (index mismatch)
I20250627 13:53:18.886621 6269 consensus_queue.cc:1035] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [LEADER]: Connected to new peer: Peer: permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 212, Time since last communication: 0.000s
I20250627 13:53:18.890190 6194 raft_consensus.cc:2953] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [term 2 LEADER]: Committing config change with OpId 2.213: config changed from index 180 to 213, 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } }
I20250627 13:53:18.890858 5370 raft_consensus.cc:2953] T 9f374c0cbcd0493597c8f30e3e63d962 P 911714a383074e09a9b2d127ab46d2c1 [term 2 FOLLOWER]: Committing config change with OpId 2.213: config changed from index 180 to 213, 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } }
I20250627 13:53:18.893857 4064 raft_consensus.cc:2953] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab [term 2 FOLLOWER]: Committing config change with OpId 2.213: config changed from index 180 to 213, 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } }
I20250627 13:53:18.904196 3788 catalog_manager.cc:5582] T 9f374c0cbcd0493597c8f30e3e63d962 P 911714a383074e09a9b2d127ab46d2c1 reported cstate change: config changed from index 180 to 213, 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "bf0be937c3634bcab62f1e58402a776f" committed_config { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } }
I20250627 13:53:18.929426 5499 consensus_queue.cc:237] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 214, Committed index: 214, Last appended: 2.215, Last appended by leader: 112, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } }
I20250627 13:53:18.930872 5370 raft_consensus.cc:1273] T 9f374c0cbcd0493597c8f30e3e63d962 P 911714a383074e09a9b2d127ab46d2c1 [term 2 FOLLOWER]: Refusing update from remote peer bf0be937c3634bcab62f1e58402a776f: Log matching property violated. Preceding OpId in replica: term: 2 index: 215. Preceding OpId from leader: term: 2 index: 216. (index mismatch)
I20250627 13:53:18.931373 6269 consensus_queue.cc:1035] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [LEADER]: Connected to new peer: Peer: permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 216, Last known committed idx: 215, Time since last communication: 0.000s
I20250627 13:53:18.935525 5370 raft_consensus.cc:2953] T 9f374c0cbcd0493597c8f30e3e63d962 P 911714a383074e09a9b2d127ab46d2c1 [term 2 FOLLOWER]: Committing config change with OpId 2.216: config changed from index 213 to 216, VOTER ad800aa8621b4fc5938692202990b5ab (127.0.103.66) evicted. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } }
I20250627 13:53:18.936978 6194 raft_consensus.cc:2953] T 9f374c0cbcd0493597c8f30e3e63d962 P bf0be937c3634bcab62f1e58402a776f [term 2 LEADER]: Committing config change with OpId 2.216: config changed from index 213 to 216, VOTER ad800aa8621b4fc5938692202990b5ab (127.0.103.66) evicted. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } }
W20250627 13:53:18.937592 4129 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.68:45503, user_credentials={real_user=slave}} blocked reactor thread for 49329.9us
I20250627 13:53:18.944273 3788 catalog_manager.cc:5582] T 9f374c0cbcd0493597c8f30e3e63d962 P 911714a383074e09a9b2d127ab46d2c1 reported cstate change: config changed from index 213 to 216, VOTER ad800aa8621b4fc5938692202990b5ab (127.0.103.66) evicted. New cstate: current_term: 2 leader_uuid: "bf0be937c3634bcab62f1e58402a776f" committed_config { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } }
I20250627 13:53:18.944581 3779 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9f374c0cbcd0493597c8f30e3e63d962 with cas_config_opid_index 213: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:53:18.954164 6060 raft_consensus.cc:1062] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6: attempting to promote NON_VOTER bf0be937c3634bcab62f1e58402a776f to VOTER
I20250627 13:53:18.958010 6060 consensus_queue.cc:237] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 202, Committed index: 202, Last appended: 1.203, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } }
I20250627 13:53:18.960646 4042 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f374c0cbcd0493597c8f30e3e63d962 with delete_type TABLET_DATA_TOMBSTONED (TS ad800aa8621b4fc5938692202990b5ab not found in new config with opid_index 216) from {username='slave'} at 127.0.0.1:33136
I20250627 13:53:18.962931 5499 raft_consensus.cc:1273] T e79731c0edcc4c11a573cf46f0b2c06a P bf0be937c3634bcab62f1e58402a776f [term 1 LEARNER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 203. Preceding OpId from leader: term: 1 index: 204. (index mismatch)
I20250627 13:53:18.963681 6343 consensus_queue.cc:1035] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 204, Last known committed idx: 202, Time since last communication: 0.000s
I20250627 13:53:18.965370 5240 raft_consensus.cc:1273] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 203. Preceding OpId from leader: term: 1 index: 204. (index mismatch)
I20250627 13:53:18.966979 6343 raft_consensus.cc:2953] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.204: config changed from index 179 to 204, bf0be937c3634bcab62f1e58402a776f (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } }
I20250627 13:53:18.969034 6341 consensus_queue.cc:1035] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 204, Last known committed idx: 202, Time since last communication: 0.000s
I20250627 13:53:18.972910 6458 tablet_replica.cc:331] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab: stopping tablet replica
I20250627 13:53:18.972674 5499 raft_consensus.cc:2953] T e79731c0edcc4c11a573cf46f0b2c06a P bf0be937c3634bcab62f1e58402a776f [term 1 FOLLOWER]: Committing config change with OpId 1.204: config changed from index 179 to 204, bf0be937c3634bcab62f1e58402a776f (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } }
I20250627 13:53:18.973112 6458 raft_consensus.cc:2241] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:53:18.973299 6458 pending_rounds.cc:70] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab: Trying to abort 1 pending ops.
I20250627 13:53:18.974905 5240 raft_consensus.cc:2953] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c [term 1 FOLLOWER]: Committing config change with OpId 1.204: config changed from index 179 to 204, bf0be937c3634bcab62f1e58402a776f (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } }
I20250627 13:53:18.973361 6458 pending_rounds.cc:77] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab: Aborting op as it isn't in flight: id { term: 2 index: 215 } timestamp: 7172228705994907648 op_type: WRITE_OP write_request { tablet_id: "9f374c0cbcd0493597c8f30e3e63d962" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000}\214zLW\345\277\215\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""8a12ce5d315481ab" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7172228705702481920 authz_token { token_data: "\010\356\306\372\302\006\"3\n\005slave\022*\n 59a8f27f17c04730859e43c3fde21ee6\020\001\030\001 \001(\001" signature: "<redacted>""v\0071\234?\225\203\377\001\316\\\300\315\224\316\234x\034\217\250\337d3\300\342\032\330\025;\322\237P\306/YK\273\005\334\330\315jR\212\003\341\366r\206\271\230\326A\202\316w\237\351\373\325\210A#\351" signing_key_seq_num: 0 } } request_id { client_id: "5acd49023c7a44ebaa9b1a29c044f133" seq_no: 557 first_incomplete_seq_no: 557 attempt_no: 0 }
I20250627 13:53:18.984275 6458 raft_consensus.cc:2270] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:53:18.985503 6458 ts_tablet_manager.cc:1905] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:53:18.989202 3788 catalog_manager.cc:5582] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 reported cstate change: config changed from index 179 to 204, bf0be937c3634bcab62f1e58402a776f (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:53:18.991232 6341 raft_consensus.cc:1062] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: attempting to promote NON_VOTER 911714a383074e09a9b2d127ab46d2c1 to VOTER
W20250627 13:53:18.993484 6343 consensus_queue.cc:1162] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "911714a383074e09a9b2d127ab46d2c1" responder_term: 1 status { last_received { term: 1 index: 197 } last_committed_idx: 197 last_received_current_leader { term: 1 index: 197 } }
I20250627 13:53:18.997555 6341 consensus_queue.cc:237] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 197, Committed index: 197, Last appended: 1.197, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 198 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } }
I20250627 13:53:18.998207 6458 ts_tablet_manager.cc:1918] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.215
I20250627 13:53:18.998385 6458 log.cc:1199] T 9f374c0cbcd0493597c8f30e3e63d962 P ad800aa8621b4fc5938692202990b5ab: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/wals/9f374c0cbcd0493597c8f30e3e63d962
I20250627 13:53:18.999406 3777 catalog_manager.cc:4928] TS ad800aa8621b4fc5938692202990b5ab (127.0.103.66:43273): tablet 9f374c0cbcd0493597c8f30e3e63d962 (table rebalance_test_table_0 [id=59a8f27f17c04730859e43c3fde21ee6]) successfully deleted
I20250627 13:53:18.999781 5372 raft_consensus.cc:1273] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1 [term 1 LEARNER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 197. Preceding OpId from leader: term: 1 index: 198. (index mismatch)
I20250627 13:53:19.000375 5241 raft_consensus.cc:1273] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 197. Preceding OpId from leader: term: 1 index: 198. (index mismatch)
I20250627 13:53:19.001003 6341 consensus_queue.cc:1035] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 198, Last known committed idx: 197, Time since last communication: 0.000s
I20250627 13:53:19.004644 6062 raft_consensus.cc:2953] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.198: config changed from index 167 to 198, 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 198 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } }
I20250627 13:53:19.005616 5370 raft_consensus.cc:2953] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1 [term 1 FOLLOWER]: Committing config change with OpId 1.198: config changed from index 167 to 198, 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 198 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } }
I20250627 13:53:19.009783 3788 catalog_manager.cc:5582] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 reported cstate change: config changed from index 167 to 198, 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 198 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:53:19.011996 6341 consensus_queue.cc:1035] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 198, Last known committed idx: 197, Time since last communication: 0.000s
I20250627 13:53:19.019455 5241 raft_consensus.cc:2953] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c [term 1 FOLLOWER]: Committing config change with OpId 1.198: config changed from index 167 to 198, 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 198 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } }
I20250627 13:53:19.097586 4194 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "8fe13ad746124b1d82cfb8ce691828a4"
dest_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54614
I20250627 13:53:19.097865 4194 raft_consensus.cc:604] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Received request to transfer leadership
I20250627 13:53:19.107566 6063 raft_consensus.cc:991] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: : Instructing follower 911714a383074e09a9b2d127ab46d2c1 to start an election
I20250627 13:53:19.107779 6063 raft_consensus.cc:1079] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Signalling peer 911714a383074e09a9b2d127ab46d2c1 to start an election
I20250627 13:53:19.108886 5371 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "8fe13ad746124b1d82cfb8ce691828a4"
dest_uuid: "911714a383074e09a9b2d127ab46d2c1"
from {username='slave'} at 127.0.103.67:52885
I20250627 13:53:19.109535 5371 raft_consensus.cc:491] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:53:19.109848 5371 raft_consensus.cc:3058] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:53:19.118665 5371 raft_consensus.cc:513] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 198 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } }
I20250627 13:53:19.121320 4194 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8fe13ad746124b1d82cfb8ce691828a4" candidate_uuid: "911714a383074e09a9b2d127ab46d2c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 201 } } ignore_live_leader: true dest_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6"
I20250627 13:53:19.121498 4194 raft_consensus.cc:3053] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Stepping down as leader of term 1
I20250627 13:53:19.121589 4194 raft_consensus.cc:738] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb586b8cb7ab44e1a7ec5016dbd2c0a6, State: Running, Role: LEADER
I20250627 13:53:19.121865 4194 consensus_queue.cc:260] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 201, Committed index: 201, Last appended: 1.201, Last appended by leader: 201, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 198 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } }
I20250627 13:53:19.122151 4194 raft_consensus.cc:3058] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:53:19.122218 5371 leader_election.cc:290] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1 [CANDIDATE]: Term 2 election: Requested vote from peers fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67:44627), 36100389495d4971b10cf590bc82c42c (127.0.103.68:45503)
I20250627 13:53:19.125399 4194 raft_consensus.cc:2466] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 911714a383074e09a9b2d127ab46d2c1 in term 2.
I20250627 13:53:19.126240 5308 leader_election.cc:304] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 911714a383074e09a9b2d127ab46d2c1, fb586b8cb7ab44e1a7ec5016dbd2c0a6; no voters:
I20250627 13:53:19.126590 6425 raft_consensus.cc:2802] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1 [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:53:19.127041 6425 raft_consensus.cc:695] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1 [term 2 LEADER]: Becoming Leader. State: Replica: 911714a383074e09a9b2d127ab46d2c1, State: Running, Role: LEADER
I20250627 13:53:19.127552 6425 consensus_queue.cc:237] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 201, Committed index: 201, Last appended: 1.201, Last appended by leader: 201, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 198 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } }
I20250627 13:53:19.137130 3788 catalog_manager.cc:5582] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1 reported cstate change: term changed from 1 to 2, leader changed from fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67) to 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69). New cstate: current_term: 2 leader_uuid: "911714a383074e09a9b2d127ab46d2c1" committed_config { opid_index: 198 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:53:19.161357 5240 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8fe13ad746124b1d82cfb8ce691828a4" candidate_uuid: "911714a383074e09a9b2d127ab46d2c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 201 } } ignore_live_leader: true dest_uuid: "36100389495d4971b10cf590bc82c42c"
I20250627 13:53:19.161692 5240 raft_consensus.cc:3058] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c [term 1 FOLLOWER]: Advancing to term 2
W20250627 13:53:19.167554 4154 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53182: Illegal state: replica fb586b8cb7ab44e1a7ec5016dbd2c0a6 is not leader of this config: current role FOLLOWER
I20250627 13:53:19.171152 5240 raft_consensus.cc:2466] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 911714a383074e09a9b2d127ab46d2c1 in term 2.
W20250627 13:53:19.241827 5736 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.0.103.126:39869, user_credentials={real_user=slave}} blocked reactor thread for 61763.1us
I20250627 13:53:19.268954 5240 raft_consensus.cc:1273] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c [term 2 FOLLOWER]: Refusing update from remote peer 911714a383074e09a9b2d127ab46d2c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 201. Preceding OpId from leader: term: 2 index: 203. (index mismatch)
I20250627 13:53:19.269153 4194 raft_consensus.cc:1273] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 2 FOLLOWER]: Refusing update from remote peer 911714a383074e09a9b2d127ab46d2c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 201. Preceding OpId from leader: term: 2 index: 203. (index mismatch)
I20250627 13:53:19.270140 6472 consensus_queue.cc:1035] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 202, Last known committed idx: 201, Time since last communication: 0.000s
I20250627 13:53:19.270777 6472 consensus_queue.cc:1035] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 202, Last known committed idx: 201, Time since last communication: 0.000s
I20250627 13:53:19.275820 6437 mvcc.cc:204] Tried to move back new op lower bound from 7172228707399249920 to 7172228706831704064. Current Snapshot: MvccSnapshot[applied={T|T < 7172228707399249920 or (T in {7172228707399249920})}]
I20250627 13:53:19.348634 5371 consensus_queue.cc:237] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 203, Committed index: 203, Last appended: 2.203, Last appended by leader: 201, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } }
I20250627 13:53:19.350595 5241 raft_consensus.cc:1273] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c [term 2 FOLLOWER]: Refusing update from remote peer 911714a383074e09a9b2d127ab46d2c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 203. Preceding OpId from leader: term: 2 index: 204. (index mismatch)
I20250627 13:53:19.351208 6472 consensus_queue.cc:1035] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 204, Last known committed idx: 203, Time since last communication: 0.000s
I20250627 13:53:19.354852 6425 raft_consensus.cc:2953] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1 [term 2 LEADER]: Committing config change with OpId 2.204: config changed from index 198 to 204, VOTER fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67) evicted. New config: { opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } }
I20250627 13:53:19.361826 3777 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 8fe13ad746124b1d82cfb8ce691828a4 with cas_config_opid_index 198: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:53:19.362438 5241 raft_consensus.cc:2953] T 8fe13ad746124b1d82cfb8ce691828a4 P 36100389495d4971b10cf590bc82c42c [term 2 FOLLOWER]: Committing config change with OpId 2.204: config changed from index 198 to 204, VOTER fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67) evicted. New config: { opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } }
I20250627 13:53:19.363328 3788 catalog_manager.cc:5582] T 8fe13ad746124b1d82cfb8ce691828a4 P 911714a383074e09a9b2d127ab46d2c1 reported cstate change: config changed from index 198 to 204, VOTER fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67) evicted. New cstate: current_term: 2 leader_uuid: "911714a383074e09a9b2d127ab46d2c1" committed_config { opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:53:19.370609 4174 tablet_service.cc:1515] Processing DeleteTablet for tablet 8fe13ad746124b1d82cfb8ce691828a4 with delete_type TABLET_DATA_TOMBSTONED (TS fb586b8cb7ab44e1a7ec5016dbd2c0a6 not found in new config with opid_index 204) from {username='slave'} at 127.0.0.1:32882
I20250627 13:53:19.395845 6482 tablet_replica.cc:331] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: stopping tablet replica
I20250627 13:53:19.397372 6482 raft_consensus.cc:2241] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:53:19.397670 6482 raft_consensus.cc:2270] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:53:19.398890 6482 ts_tablet_manager.cc:1905] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:53:19.406414 6482 ts_tablet_manager.cc:1918] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.203
I20250627 13:53:19.406646 6482 log.cc:1199] T 8fe13ad746124b1d82cfb8ce691828a4 P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal/wals/8fe13ad746124b1d82cfb8ce691828a4
I20250627 13:53:19.407516 3779 catalog_manager.cc:4928] TS fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67:44627): tablet 8fe13ad746124b1d82cfb8ce691828a4 (table rebalance_test_table_4 [id=b5d412773a2343d1919352385a3629db]) successfully deleted
I20250627 13:53:19.440886 4194 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "e79731c0edcc4c11a573cf46f0b2c06a"
dest_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54624
I20250627 13:53:19.441133 4194 raft_consensus.cc:604] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Received request to transfer leadership
I20250627 13:53:19.608568 6062 raft_consensus.cc:991] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6: : Instructing follower 36100389495d4971b10cf590bc82c42c to start an election
I20250627 13:53:19.608789 6062 raft_consensus.cc:1079] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Signalling peer 36100389495d4971b10cf590bc82c42c to start an election
I20250627 13:53:19.609895 5241 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "e79731c0edcc4c11a573cf46f0b2c06a"
dest_uuid: "36100389495d4971b10cf590bc82c42c"
from {username='slave'} at 127.0.103.67:46935
I20250627 13:53:19.610289 5241 raft_consensus.cc:491] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:53:19.610461 5241 raft_consensus.cc:3058] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:53:19.648278 5241 raft_consensus.cc:513] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } }
I20250627 13:53:19.651031 5241 leader_election.cc:290] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c [CANDIDATE]: Term 2 election: Requested vote from peers fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67:44627), bf0be937c3634bcab62f1e58402a776f (127.0.103.70:37215)
I20250627 13:53:19.651175 4194 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e79731c0edcc4c11a573cf46f0b2c06a" candidate_uuid: "36100389495d4971b10cf590bc82c42c" candidate_term: 2 candidate_status { last_received { term: 1 index: 208 } } ignore_live_leader: true dest_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6"
I20250627 13:53:19.651391 4194 raft_consensus.cc:3053] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Stepping down as leader of term 1
I20250627 13:53:19.651468 4194 raft_consensus.cc:738] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb586b8cb7ab44e1a7ec5016dbd2c0a6, State: Running, Role: LEADER
I20250627 13:53:19.651722 4194 consensus_queue.cc:260] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 208, Committed index: 208, Last appended: 1.208, Last appended by leader: 208, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } }
I20250627 13:53:19.652037 4194 raft_consensus.cc:3058] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:53:19.658643 4194 raft_consensus.cc:2466] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36100389495d4971b10cf590bc82c42c in term 2.
I20250627 13:53:19.659238 5177 leader_election.cc:304] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36100389495d4971b10cf590bc82c42c, fb586b8cb7ab44e1a7ec5016dbd2c0a6; no voters:
I20250627 13:53:19.659641 6459 raft_consensus.cc:2802] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:53:19.674178 5499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e79731c0edcc4c11a573cf46f0b2c06a" candidate_uuid: "36100389495d4971b10cf590bc82c42c" candidate_term: 2 candidate_status { last_received { term: 1 index: 208 } } ignore_live_leader: true dest_uuid: "bf0be937c3634bcab62f1e58402a776f"
I20250627 13:53:19.674566 5499 raft_consensus.cc:3058] T e79731c0edcc4c11a573cf46f0b2c06a P bf0be937c3634bcab62f1e58402a776f [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:53:19.706785 5499 raft_consensus.cc:2466] T e79731c0edcc4c11a573cf46f0b2c06a P bf0be937c3634bcab62f1e58402a776f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36100389495d4971b10cf590bc82c42c in term 2.
I20250627 13:53:19.720513 6459 raft_consensus.cc:695] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c [term 2 LEADER]: Becoming Leader. State: Replica: 36100389495d4971b10cf590bc82c42c, State: Running, Role: LEADER
I20250627 13:53:19.721197 6459 consensus_queue.cc:237] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 208, Committed index: 208, Last appended: 1.208, Last appended by leader: 208, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } }
I20250627 13:53:19.725030 3784 catalog_manager.cc:5582] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c reported cstate change: term changed from 1 to 2, leader changed from fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67) to 36100389495d4971b10cf590bc82c42c (127.0.103.68). New cstate: current_term: 2 leader_uuid: "36100389495d4971b10cf590bc82c42c" committed_config { opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:19.846992 4194 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "e79731c0edcc4c11a573cf46f0b2c06a"
dest_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54644
I20250627 13:53:19.847236 4194 raft_consensus.cc:604] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 2 FOLLOWER]: Received request to transfer leadership
I20250627 13:53:19.847333 4194 raft_consensus.cc:612] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250627 13:53:19.891340 3869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.70:37215, user_credentials={real_user=slave}} blocked reactor thread for 63079.5us
W20250627 13:53:20.066638 4154 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:53174: Illegal state: replica fb586b8cb7ab44e1a7ec5016dbd2c0a6 is not leader of this config: current role FOLLOWER
I20250627 13:53:20.118024 5499 raft_consensus.cc:1273] T e79731c0edcc4c11a573cf46f0b2c06a P bf0be937c3634bcab62f1e58402a776f [term 2 FOLLOWER]: Refusing update from remote peer 36100389495d4971b10cf590bc82c42c: Log matching property violated. Preceding OpId in replica: term: 1 index: 208. Preceding OpId from leader: term: 2 index: 209. (index mismatch)
I20250627 13:53:20.131119 6459 consensus_queue.cc:1035] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 209, Last known committed idx: 208, Time since last communication: 0.000s
I20250627 13:53:20.133199 4194 raft_consensus.cc:1273] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 2 FOLLOWER]: Refusing update from remote peer 36100389495d4971b10cf590bc82c42c: Log matching property violated. Preceding OpId in replica: term: 1 index: 208. Preceding OpId from leader: term: 2 index: 209. (index mismatch)
I20250627 13:53:20.133793 6459 consensus_queue.cc:1035] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 209, Last known committed idx: 208, Time since last communication: 0.000s
I20250627 13:53:20.186813 5240 consensus_queue.cc:237] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 209, Committed index: 209, Last appended: 2.209, Last appended by leader: 208, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } }
I20250627 13:53:20.192049 5501 raft_consensus.cc:1273] T e79731c0edcc4c11a573cf46f0b2c06a P bf0be937c3634bcab62f1e58402a776f [term 2 FOLLOWER]: Refusing update from remote peer 36100389495d4971b10cf590bc82c42c: Log matching property violated. Preceding OpId in replica: term: 2 index: 209. Preceding OpId from leader: term: 2 index: 210. (index mismatch)
I20250627 13:53:20.205458 6459 consensus_queue.cc:1035] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 210, Last known committed idx: 209, Time since last communication: 0.000s
I20250627 13:53:20.209434 6511 raft_consensus.cc:2953] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c [term 2 LEADER]: Committing config change with OpId 2.210: config changed from index 204 to 210, VOTER fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67) evicted. New config: { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } }
I20250627 13:53:20.210144 5501 raft_consensus.cc:2953] T e79731c0edcc4c11a573cf46f0b2c06a P bf0be937c3634bcab62f1e58402a776f [term 2 FOLLOWER]: Committing config change with OpId 2.210: config changed from index 204 to 210, VOTER fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67) evicted. New config: { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } } }
I20250627 13:53:20.213325 3778 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet e79731c0edcc4c11a573cf46f0b2c06a with cas_config_opid_index 204: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:53:20.267972 3784 catalog_manager.cc:5582] T e79731c0edcc4c11a573cf46f0b2c06a P 36100389495d4971b10cf590bc82c42c reported cstate change: config changed from index 204 to 210, VOTER fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67) evicted. New cstate: current_term: 2 leader_uuid: "36100389495d4971b10cf590bc82c42c" committed_config { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "36100389495d4971b10cf590bc82c42c" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 45503 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:53:20.297180 4174 tablet_service.cc:1515] Processing DeleteTablet for tablet e79731c0edcc4c11a573cf46f0b2c06a with delete_type TABLET_DATA_TOMBSTONED (TS fb586b8cb7ab44e1a7ec5016dbd2c0a6 not found in new config with opid_index 210) from {username='slave'} at 127.0.0.1:32882
I20250627 13:53:20.322533 6542 tablet_replica.cc:331] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6: stopping tablet replica
I20250627 13:53:20.322817 6542 raft_consensus.cc:2241] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:53:20.323099 6542 raft_consensus.cc:2270] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:53:20.331336 6542 ts_tablet_manager.cc:1905] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:53:20.340005 6542 ts_tablet_manager.cc:1918] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.209
I20250627 13:53:20.340210 6542 log.cc:1199] T e79731c0edcc4c11a573cf46f0b2c06a P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal/wals/e79731c0edcc4c11a573cf46f0b2c06a
I20250627 13:53:20.341809 3779 catalog_manager.cc:4928] TS fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67:44627): tablet e79731c0edcc4c11a573cf46f0b2c06a (table rebalance_test_table_1 [id=70cec1ac0b6b4a82a338f38f22668b39]) successfully deleted
I20250627 13:53:21.278474 5221 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:21.304308 5614 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:21.306262 5352 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:21.359761 4042 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:53:21.430668 4174 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:53:21.542986 5482 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
I20250627 13:53:21.562296 3912 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
W20250627 13:53:22.360438 5567 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.68:45503, user_credentials={real_user=slave}} blocked reactor thread for 47109.9us
I20250627 13:53:22.875605 5221 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:23.003959 5352 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:23.047610 5482 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
I20250627 13:53:23.048696 4174 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:53:23.061091 5614 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:23.081393 3912 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
I20250627 13:53:23.106505 4042 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
W20250627 13:53:24.707999 5307 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.68:45503, user_credentials={real_user=slave}} blocked reactor thread for 55666.3us
I20250627 13:53:25.133684 5221 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:25.236769 5352 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:25.253711 4042 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:53:25.311761 5614 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:25.316910 3912 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
I20250627 13:53:25.326624 4174 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:53:25.419840 5482 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
I20250627 13:53:26.186303 5221 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:26.275696 5352 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:26.284193 5614 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:26.306810 5482 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
I20250627 13:53:26.332316 3912 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
I20250627 13:53:26.353965 4174 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:53:26.406154 4042 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:53:27.171587 5221 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:27.177769 3912 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
I20250627 13:53:27.182317 4174 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:53:27.202447 5482 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
I20250627 13:53:27.245072 5352 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:27.250861 5614 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:27.267163 4042 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bcc04dadc3364622b3a0e2d593b2dc60 | 127.0.103.126:39869 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.103.84:41153 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
36100389495d4971b10cf590bc82c42c | 127.0.103.68:45503 | HEALTHY | <none> | 1 | 0
6503b59977444aaebe5ee70a0bc99154 | 127.0.103.71:45413 | HEALTHY | <none> | 1 | 0
911714a383074e09a9b2d127ab46d2c1 | 127.0.103.69:42665 | HEALTHY | <none> | 1 | 0
ad800aa8621b4fc5938692202990b5ab | 127.0.103.66:43273 | HEALTHY | <none> | 4 | 0
bf0be937c3634bcab62f1e58402a776f | 127.0.103.70:37215 | HEALTHY | <none> | 2 | 0
f46fd51df5864af9b8493acd69db0747 | 127.0.103.65:35133 | HEALTHY | <none> | 2 | 0
fb586b8cb7ab44e1a7ec5016dbd2c0a6 | 127.0.103.67:44627 | HEALTHY | <none> | 5 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.0.103.65 | experimental | 127.0.103.65:35133
local_ip_for_outbound_sockets | 127.0.103.66 | experimental | 127.0.103.66:43273
local_ip_for_outbound_sockets | 127.0.103.67 | experimental | 127.0.103.67:44627
local_ip_for_outbound_sockets | 127.0.103.68 | experimental | 127.0.103.68:45503
local_ip_for_outbound_sockets | 127.0.103.69 | experimental | 127.0.103.69:42665
local_ip_for_outbound_sockets | 127.0.103.70 | experimental | 127.0.103.70:37215
local_ip_for_outbound_sockets | 127.0.103.71 | experimental | 127.0.103.71:45413
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.103.65:35133
server_dump_info_path | /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.103.66:43273
server_dump_info_path | /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.103.67:44627
server_dump_info_path | /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.103.68:45503
server_dump_info_path | /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.103.69:42665
server_dump_info_path | /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.103.70:37215
server_dump_info_path | /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.103.71:45413
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.103.84:41153 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250627 13:53:27.338754 413 log_verifier.cc:126] Checking tablet 0fdcc98da13c4cf998b1fd1b3e3f9fe5
I20250627 13:53:27.358134 413 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 0fdcc98da13c4cf998b1fd1b3e3f9fe5
I20250627 13:53:27.358273 413 log_verifier.cc:126] Checking tablet 16fed4ebf25c44069b67af98964ab075
I20250627 13:53:27.484341 413 log_verifier.cc:177] Verified matching terms for 337 ops in tablet 16fed4ebf25c44069b67af98964ab075
I20250627 13:53:27.484795 413 log_verifier.cc:126] Checking tablet 1cd6f75e462546c6bb05ef14af3498fa
I20250627 13:53:27.643358 413 log_verifier.cc:177] Verified matching terms for 367 ops in tablet 1cd6f75e462546c6bb05ef14af3498fa
I20250627 13:53:27.643983 413 log_verifier.cc:126] Checking tablet 37f6cffc2bc5456e94999b60340dfeb6
I20250627 13:53:27.816183 413 log_verifier.cc:177] Verified matching terms for 381 ops in tablet 37f6cffc2bc5456e94999b60340dfeb6
I20250627 13:53:27.816613 413 log_verifier.cc:126] Checking tablet 3e2b11c6788e4ccdbdde60769fee1aae
I20250627 13:53:27.943445 413 log_verifier.cc:177] Verified matching terms for 348 ops in tablet 3e2b11c6788e4ccdbdde60769fee1aae
I20250627 13:53:27.943867 413 log_verifier.cc:126] Checking tablet 40b04be88e9947c5a5ca8432e3791bdd
I20250627 13:53:28.079391 413 log_verifier.cc:177] Verified matching terms for 363 ops in tablet 40b04be88e9947c5a5ca8432e3791bdd
I20250627 13:53:28.079759 413 log_verifier.cc:126] Checking tablet 4a118f20daaf45a2a1c539410dbf0d30
I20250627 13:53:28.227025 413 log_verifier.cc:177] Verified matching terms for 392 ops in tablet 4a118f20daaf45a2a1c539410dbf0d30
I20250627 13:53:28.227383 413 log_verifier.cc:126] Checking tablet 76518bc59dfe443abd8e97539d1bcb8a
I20250627 13:53:28.372103 413 log_verifier.cc:177] Verified matching terms for 385 ops in tablet 76518bc59dfe443abd8e97539d1bcb8a
I20250627 13:53:28.372531 413 log_verifier.cc:126] Checking tablet 8fe13ad746124b1d82cfb8ce691828a4
I20250627 13:53:28.497651 413 log_verifier.cc:177] Verified matching terms for 325 ops in tablet 8fe13ad746124b1d82cfb8ce691828a4
I20250627 13:53:28.498030 413 log_verifier.cc:126] Checking tablet 978e20547d644d3988d6da29235903aa
I20250627 13:53:28.625581 413 log_verifier.cc:177] Verified matching terms for 317 ops in tablet 978e20547d644d3988d6da29235903aa
I20250627 13:53:28.625967 413 log_verifier.cc:126] Checking tablet 9f374c0cbcd0493597c8f30e3e63d962
I20250627 13:53:28.776381 413 log_verifier.cc:177] Verified matching terms for 394 ops in tablet 9f374c0cbcd0493597c8f30e3e63d962
I20250627 13:53:28.776764 413 log_verifier.cc:126] Checking tablet a01527b9f7c14748837f5ad4dbaa9e9d
I20250627 13:53:28.898108 413 log_verifier.cc:177] Verified matching terms for 318 ops in tablet a01527b9f7c14748837f5ad4dbaa9e9d
I20250627 13:53:28.898470 413 log_verifier.cc:126] Checking tablet b2b66876e09748b49f27128fe7591be9
I20250627 13:53:29.033315 413 log_verifier.cc:177] Verified matching terms for 348 ops in tablet b2b66876e09748b49f27128fe7591be9
I20250627 13:53:29.033712 413 log_verifier.cc:126] Checking tablet be42f2b9b1b747adb641149f70da608a
I20250627 13:53:29.183311 413 log_verifier.cc:177] Verified matching terms for 339 ops in tablet be42f2b9b1b747adb641149f70da608a
I20250627 13:53:29.184051 413 log_verifier.cc:126] Checking tablet c20692dbde954a259acdb9c765978113
I20250627 13:53:29.358593 413 log_verifier.cc:177] Verified matching terms for 329 ops in tablet c20692dbde954a259acdb9c765978113
I20250627 13:53:29.358974 413 log_verifier.cc:126] Checking tablet e79731c0edcc4c11a573cf46f0b2c06a
I20250627 13:53:29.485394 413 log_verifier.cc:177] Verified matching terms for 341 ops in tablet e79731c0edcc4c11a573cf46f0b2c06a
I20250627 13:53:29.488493 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.72:0
--local_ip_for_outbound_sockets=127.0.103.72
--webserver_interface=127.0.103.72
--webserver_port=0
--tserver_master_addrs=127.0.103.126:39869
--builtin_ntp_servers=127.0.103.84:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250627 13:53:29.838618 6860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:53:29.838961 6860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:53:29.839120 6860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:53:29.846681 6860 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:53:29.846776 6860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:53:29.846905 6860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.72
I20250627 13:53:29.856159 6860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.103.72
--webserver_port=0
--tserver_master_addrs=127.0.103.126:39869
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.72
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:53:29.857662 6860 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:53:29.859344 6860 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:53:29.870337 6867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:53:29.870478 6869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:53:29.870873 6866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:53:29.871636 6860 server_base.cc:1048] running on GCE node
I20250627 13:53:29.872151 6860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:53:29.872952 6860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:53:29.874164 6860 hybrid_clock.cc:648] HybridClock initialized: now 1751032409874109 us; error 37 us; skew 500 ppm
I20250627 13:53:29.874409 6860 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:53:29.876436 6860 webserver.cc:469] Webserver started at http://127.0.103.72:41755/ using document root <none> and password file <none>
I20250627 13:53:29.877061 6860 fs_manager.cc:362] Metadata directory not provided
I20250627 13:53:29.877174 6860 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:53:29.877462 6860 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:53:29.879884 6860 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "158cc55ab37e4862bb0a908b720e94e3"
format_stamp: "Formatted at 2025-06-27 13:53:29 on dist-test-slave-1ckg"
I20250627 13:53:29.881052 6860 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "158cc55ab37e4862bb0a908b720e94e3"
format_stamp: "Formatted at 2025-06-27 13:53:29 on dist-test-slave-1ckg"
I20250627 13:53:29.885573 6860 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20250627 13:53:29.888551 6875 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:29.889672 6860 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250627 13:53:29.889814 6860 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "158cc55ab37e4862bb0a908b720e94e3"
format_stamp: "Formatted at 2025-06-27 13:53:29 on dist-test-slave-1ckg"
I20250627 13:53:29.889981 6860 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:53:29.910141 6860 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:53:29.910914 6860 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:53:29.911171 6860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:53:29.912012 6860 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:53:29.913498 6860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:53:29.913595 6860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:29.913731 6860 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:53:29.913787 6860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:29.947286 6860 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.72:45709
I20250627 13:53:29.947341 6987 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.72:45709 every 8 connection(s)
I20250627 13:53:29.948603 6860 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20250627 13:53:29.952688 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 6860
I20250627 13:53:29.952883 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20250627 13:53:29.965919 6988 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39869
I20250627 13:53:29.966307 6988 heartbeater.cc:461] Registering TS with master...
I20250627 13:53:29.967319 6988 heartbeater.cc:507] Master 127.0.103.126:39869 requested a full tablet report, sending...
I20250627 13:53:29.968865 3789 ts_manager.cc:194] Registered new tserver with Master: 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72:45709)
I20250627 13:53:29.969831 3789 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.72:59181
I20250627 13:53:30.448536 6923 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:53:30.455358 5221 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:30.463121 5614 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:30.474998 5482 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
I20250627 13:53:30.510097 3912 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
I20250627 13:53:30.510563 4042 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:53:30.528036 4174 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:53:30.551570 5352 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:30.734336 6923 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:53:30.740087 5221 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:30.756543 5482 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
I20250627 13:53:30.768466 5614 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:30.769480 3912 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
I20250627 13:53:30.770383 4042 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:53:30.781003 4174 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:53:30.819489 5352 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:30.887161 4194 consensus_queue.cc:237] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 367, Committed index: 367, Last appended: 1.367, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: true } }
I20250627 13:53:30.889820 5369 raft_consensus.cc:1273] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1 [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 367. Preceding OpId from leader: term: 1 index: 368. (index mismatch)
I20250627 13:53:30.890370 7085 consensus_queue.cc:1035] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 368, Last known committed idx: 367, Time since last communication: 0.000s
I20250627 13:53:30.893141 6808 raft_consensus.cc:2953] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.368: config changed from index 107 to 368, NON_VOTER 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72) added. New config: { opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: true } } }
I20250627 13:53:30.893630 5369 raft_consensus.cc:2953] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1 [term 1 FOLLOWER]: Committing config change with OpId 1.368: config changed from index 107 to 368, NON_VOTER 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72) added. New config: { opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: true } } }
I20250627 13:53:30.897938 3790 catalog_manager.cc:5582] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 reported cstate change: config changed from index 107 to 368, NON_VOTER 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72) added. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250627 13:53:30.899250 4128 consensus_peers.cc:487] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 -> Peer 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72:45709): Couldn't send request to peer 158cc55ab37e4862bb0a908b720e94e3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1cd6f75e462546c6bb05ef14af3498fa. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:53:30.911065 4064 consensus_queue.cc:237] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 318, Committed index: 318, Last appended: 1.318, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: true } }
I20250627 13:53:30.913308 3930 raft_consensus.cc:1273] T a01527b9f7c14748837f5ad4dbaa9e9d P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Refusing update from remote peer ad800aa8621b4fc5938692202990b5ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 318. Preceding OpId from leader: term: 1 index: 319. (index mismatch)
I20250627 13:53:30.913949 6606 consensus_queue.cc:1035] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 319, Last known committed idx: 318, Time since last communication: 0.000s
I20250627 13:53:30.916597 7098 raft_consensus.cc:2953] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab [term 1 LEADER]: Committing config change with OpId 1.319: config changed from index -1 to 319, NON_VOTER 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72) added. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: true } } }
I20250627 13:53:30.917475 3930 raft_consensus.cc:2953] T a01527b9f7c14748837f5ad4dbaa9e9d P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Committing config change with OpId 1.319: config changed from index -1 to 319, NON_VOTER 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72) added. New config: { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: true } } }
I20250627 13:53:30.922389 3789 catalog_manager.cc:5582] T a01527b9f7c14748837f5ad4dbaa9e9d P f46fd51df5864af9b8493acd69db0747 reported cstate change: config changed from index -1 to 319, NON_VOTER 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72) added. New cstate: current_term: 1 leader_uuid: "ad800aa8621b4fc5938692202990b5ab" committed_config { opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: true } } }
W20250627 13:53:30.926416 3998 consensus_peers.cc:487] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab -> Peer 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72:45709): Couldn't send request to peer 158cc55ab37e4862bb0a908b720e94e3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a01527b9f7c14748837f5ad4dbaa9e9d. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:53:30.934847 4064 consensus_queue.cc:237] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 381, Committed index: 381, Last appended: 1.381, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: true } }
I20250627 13:53:30.936803 5369 raft_consensus.cc:1273] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1 [term 1 FOLLOWER]: Refusing update from remote peer ad800aa8621b4fc5938692202990b5ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 381. Preceding OpId from leader: term: 1 index: 382. (index mismatch)
W20250627 13:53:30.936955 3998 consensus_peers.cc:487] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab -> Peer 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72:45709): Couldn't send request to peer 158cc55ab37e4862bb0a908b720e94e3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37f6cffc2bc5456e94999b60340dfeb6. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:53:30.937247 7098 consensus_queue.cc:1035] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 382, Last known committed idx: 381, Time since last communication: 0.000s
I20250627 13:53:30.939523 6606 raft_consensus.cc:2953] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [term 1 LEADER]: Committing config change with OpId 1.382: config changed from index 111 to 382, NON_VOTER 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72) added. New config: { opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: true } } }
I20250627 13:53:30.940057 5369 raft_consensus.cc:2953] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1 [term 1 FOLLOWER]: Committing config change with OpId 1.382: config changed from index 111 to 382, NON_VOTER 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72) added. New config: { opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: true } } }
I20250627 13:53:30.944116 3790 catalog_manager.cc:5582] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1 reported cstate change: config changed from index 111 to 382, NON_VOTER 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72) added. New cstate: current_term: 1 leader_uuid: "ad800aa8621b4fc5938692202990b5ab" committed_config { opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: true } } }
I20250627 13:53:30.955339 4194 consensus_queue.cc:237] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 339, Committed index: 339, Last appended: 1.339, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false replace: true } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: true } }
W20250627 13:53:30.958084 4128 consensus_peers.cc:487] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 -> Peer 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72:45709): Couldn't send request to peer 158cc55ab37e4862bb0a908b720e94e3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: be42f2b9b1b747adb641149f70da608a. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:53:30.958086 5632 raft_consensus.cc:1273] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154 [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 339. Preceding OpId from leader: term: 1 index: 340. (index mismatch)
I20250627 13:53:30.958585 7085 consensus_queue.cc:1035] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 340, Last known committed idx: 339, Time since last communication: 0.000s
I20250627 13:53:30.960937 6808 raft_consensus.cc:2953] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.340: config changed from index 123 to 340, NON_VOTER 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72) added. New config: { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false replace: true } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: true } } }
I20250627 13:53:30.961488 5632 raft_consensus.cc:2953] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154 [term 1 FOLLOWER]: Committing config change with OpId 1.340: config changed from index 123 to 340, NON_VOTER 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72) added. New config: { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false replace: true } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: true } } }
I20250627 13:53:30.965165 3789 catalog_manager.cc:5582] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 reported cstate change: config changed from index 123 to 340, NON_VOTER 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72) added. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:30.972716 6988 heartbeater.cc:499] Master 127.0.103.126:39869 was elected leader, sending a full tablet report...
I20250627 13:53:31.415699 7156 ts_tablet_manager.cc:927] T be42f2b9b1b747adb641149f70da608a P 158cc55ab37e4862bb0a908b720e94e3: Initiating tablet copy from peer fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67:44627)
I20250627 13:53:31.416626 7156 tablet_copy_client.cc:323] T be42f2b9b1b747adb641149f70da608a P 158cc55ab37e4862bb0a908b720e94e3: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:44627
I20250627 13:53:31.422995 4213 tablet_copy_service.cc:140] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Received BeginTabletCopySession request for tablet be42f2b9b1b747adb641149f70da608a from peer 158cc55ab37e4862bb0a908b720e94e3 ({username='slave'} at 127.0.103.72:39721)
I20250627 13:53:31.423125 4213 tablet_copy_service.cc:161] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Beginning new tablet copy session on tablet be42f2b9b1b747adb641149f70da608a from peer 158cc55ab37e4862bb0a908b720e94e3 at {username='slave'} at 127.0.103.72:39721: session id = 158cc55ab37e4862bb0a908b720e94e3-be42f2b9b1b747adb641149f70da608a
I20250627 13:53:31.425122 4213 tablet_copy_source_session.cc:215] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:53:31.429513 7156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be42f2b9b1b747adb641149f70da608a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:31.439332 7156 tablet_copy_client.cc:806] T be42f2b9b1b747adb641149f70da608a P 158cc55ab37e4862bb0a908b720e94e3: tablet copy: Starting download of 0 data blocks...
I20250627 13:53:31.439610 7156 tablet_copy_client.cc:670] T be42f2b9b1b747adb641149f70da608a P 158cc55ab37e4862bb0a908b720e94e3: tablet copy: Starting download of 1 WAL segments...
I20250627 13:53:31.441138 7159 ts_tablet_manager.cc:927] T a01527b9f7c14748837f5ad4dbaa9e9d P 158cc55ab37e4862bb0a908b720e94e3: Initiating tablet copy from peer ad800aa8621b4fc5938692202990b5ab (127.0.103.66:43273)
I20250627 13:53:31.441812 7159 tablet_copy_client.cc:323] T a01527b9f7c14748837f5ad4dbaa9e9d P 158cc55ab37e4862bb0a908b720e94e3: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:43273
I20250627 13:53:31.443223 7156 tablet_copy_client.cc:538] T be42f2b9b1b747adb641149f70da608a P 158cc55ab37e4862bb0a908b720e94e3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:53:31.446960 7156 tablet_bootstrap.cc:492] T be42f2b9b1b747adb641149f70da608a P 158cc55ab37e4862bb0a908b720e94e3: Bootstrap starting.
I20250627 13:53:31.449824 4084 tablet_copy_service.cc:140] P ad800aa8621b4fc5938692202990b5ab: Received BeginTabletCopySession request for tablet a01527b9f7c14748837f5ad4dbaa9e9d from peer 158cc55ab37e4862bb0a908b720e94e3 ({username='slave'} at 127.0.103.72:45649)
I20250627 13:53:31.449971 4084 tablet_copy_service.cc:161] P ad800aa8621b4fc5938692202990b5ab: Beginning new tablet copy session on tablet a01527b9f7c14748837f5ad4dbaa9e9d from peer 158cc55ab37e4862bb0a908b720e94e3 at {username='slave'} at 127.0.103.72:45649: session id = 158cc55ab37e4862bb0a908b720e94e3-a01527b9f7c14748837f5ad4dbaa9e9d
I20250627 13:53:31.452529 4084 tablet_copy_source_session.cc:215] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:53:31.453943 7159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a01527b9f7c14748837f5ad4dbaa9e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:31.460506 7159 tablet_copy_client.cc:806] T a01527b9f7c14748837f5ad4dbaa9e9d P 158cc55ab37e4862bb0a908b720e94e3: tablet copy: Starting download of 0 data blocks...
I20250627 13:53:31.460826 7159 tablet_copy_client.cc:670] T a01527b9f7c14748837f5ad4dbaa9e9d P 158cc55ab37e4862bb0a908b720e94e3: tablet copy: Starting download of 1 WAL segments...
I20250627 13:53:31.463930 7159 tablet_copy_client.cc:538] T a01527b9f7c14748837f5ad4dbaa9e9d P 158cc55ab37e4862bb0a908b720e94e3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:53:31.465476 7172 ts_tablet_manager.cc:927] T 1cd6f75e462546c6bb05ef14af3498fa P 158cc55ab37e4862bb0a908b720e94e3: Initiating tablet copy from peer fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67:44627)
I20250627 13:53:31.468761 7172 tablet_copy_client.cc:323] T 1cd6f75e462546c6bb05ef14af3498fa P 158cc55ab37e4862bb0a908b720e94e3: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:44627
I20250627 13:53:31.469225 7159 tablet_bootstrap.cc:492] T a01527b9f7c14748837f5ad4dbaa9e9d P 158cc55ab37e4862bb0a908b720e94e3: Bootstrap starting.
I20250627 13:53:31.470201 4213 tablet_copy_service.cc:140] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Received BeginTabletCopySession request for tablet 1cd6f75e462546c6bb05ef14af3498fa from peer 158cc55ab37e4862bb0a908b720e94e3 ({username='slave'} at 127.0.103.72:39721)
I20250627 13:53:31.470419 4213 tablet_copy_service.cc:161] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Beginning new tablet copy session on tablet 1cd6f75e462546c6bb05ef14af3498fa from peer 158cc55ab37e4862bb0a908b720e94e3 at {username='slave'} at 127.0.103.72:39721: session id = 158cc55ab37e4862bb0a908b720e94e3-1cd6f75e462546c6bb05ef14af3498fa
I20250627 13:53:31.473078 4213 tablet_copy_source_session.cc:215] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:53:31.474512 7172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cd6f75e462546c6bb05ef14af3498fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:31.484830 7172 tablet_copy_client.cc:806] T 1cd6f75e462546c6bb05ef14af3498fa P 158cc55ab37e4862bb0a908b720e94e3: tablet copy: Starting download of 0 data blocks...
I20250627 13:53:31.485371 7172 tablet_copy_client.cc:670] T 1cd6f75e462546c6bb05ef14af3498fa P 158cc55ab37e4862bb0a908b720e94e3: tablet copy: Starting download of 1 WAL segments...
I20250627 13:53:31.488719 7172 tablet_copy_client.cc:538] T 1cd6f75e462546c6bb05ef14af3498fa P 158cc55ab37e4862bb0a908b720e94e3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:53:31.495380 7172 tablet_bootstrap.cc:492] T 1cd6f75e462546c6bb05ef14af3498fa P 158cc55ab37e4862bb0a908b720e94e3: Bootstrap starting.
I20250627 13:53:31.534005 7156 log.cc:826] T be42f2b9b1b747adb641149f70da608a P 158cc55ab37e4862bb0a908b720e94e3: Log is configured to *not* fsync() on all Append() calls
I20250627 13:53:31.575743 7186 ts_tablet_manager.cc:927] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3: Initiating tablet copy from peer ad800aa8621b4fc5938692202990b5ab (127.0.103.66:43273)
I20250627 13:53:31.608376 7186 tablet_copy_client.cc:323] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:43273
I20250627 13:53:31.609259 4084 tablet_copy_service.cc:140] P ad800aa8621b4fc5938692202990b5ab: Received BeginTabletCopySession request for tablet 37f6cffc2bc5456e94999b60340dfeb6 from peer 158cc55ab37e4862bb0a908b720e94e3 ({username='slave'} at 127.0.103.72:45649)
I20250627 13:53:31.609462 4084 tablet_copy_service.cc:161] P ad800aa8621b4fc5938692202990b5ab: Beginning new tablet copy session on tablet 37f6cffc2bc5456e94999b60340dfeb6 from peer 158cc55ab37e4862bb0a908b720e94e3 at {username='slave'} at 127.0.103.72:45649: session id = 158cc55ab37e4862bb0a908b720e94e3-37f6cffc2bc5456e94999b60340dfeb6
I20250627 13:53:31.612668 4084 tablet_copy_source_session.cc:215] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:53:31.614137 7186 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37f6cffc2bc5456e94999b60340dfeb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:31.621423 7186 tablet_copy_client.cc:806] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3: tablet copy: Starting download of 0 data blocks...
I20250627 13:53:31.621743 7186 tablet_copy_client.cc:670] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3: tablet copy: Starting download of 1 WAL segments...
I20250627 13:53:31.625238 7186 tablet_copy_client.cc:538] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:53:31.628995 7186 tablet_bootstrap.cc:492] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3: Bootstrap starting.
I20250627 13:53:31.925026 7159 tablet_bootstrap.cc:492] T a01527b9f7c14748837f5ad4dbaa9e9d P 158cc55ab37e4862bb0a908b720e94e3: Bootstrap replayed 1/1 log segments. Stats: ops{read=319 overwritten=0 applied=319 ignored=0} inserts{seen=991 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:53:31.925704 7159 tablet_bootstrap.cc:492] T a01527b9f7c14748837f5ad4dbaa9e9d P 158cc55ab37e4862bb0a908b720e94e3: Bootstrap complete.
I20250627 13:53:31.926301 7159 ts_tablet_manager.cc:1397] T a01527b9f7c14748837f5ad4dbaa9e9d P 158cc55ab37e4862bb0a908b720e94e3: Time spent bootstrapping tablet: real 0.457s user 0.361s sys 0.064s
I20250627 13:53:31.933873 7159 raft_consensus.cc:357] T a01527b9f7c14748837f5ad4dbaa9e9d P 158cc55ab37e4862bb0a908b720e94e3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: true } }
I20250627 13:53:31.934391 7159 raft_consensus.cc:738] T a01527b9f7c14748837f5ad4dbaa9e9d P 158cc55ab37e4862bb0a908b720e94e3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 158cc55ab37e4862bb0a908b720e94e3, State: Initialized, Role: LEARNER
I20250627 13:53:31.934958 7159 consensus_queue.cc:260] T a01527b9f7c14748837f5ad4dbaa9e9d P 158cc55ab37e4862bb0a908b720e94e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 319, Last appended: 1.319, Last appended by leader: 319, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 319 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: true } }
I20250627 13:53:31.950569 7159 ts_tablet_manager.cc:1428] T a01527b9f7c14748837f5ad4dbaa9e9d P 158cc55ab37e4862bb0a908b720e94e3: Time spent starting tablet: real 0.024s user 0.005s sys 0.003s
I20250627 13:53:31.951390 4084 tablet_copy_service.cc:342] P ad800aa8621b4fc5938692202990b5ab: Request end of tablet copy session 158cc55ab37e4862bb0a908b720e94e3-a01527b9f7c14748837f5ad4dbaa9e9d received from {username='slave'} at 127.0.103.72:45649
I20250627 13:53:31.951543 4084 tablet_copy_service.cc:434] P ad800aa8621b4fc5938692202990b5ab: ending tablet copy session 158cc55ab37e4862bb0a908b720e94e3-a01527b9f7c14748837f5ad4dbaa9e9d on tablet a01527b9f7c14748837f5ad4dbaa9e9d with peer 158cc55ab37e4862bb0a908b720e94e3
I20250627 13:53:32.045012 7156 tablet_bootstrap.cc:492] T be42f2b9b1b747adb641149f70da608a P 158cc55ab37e4862bb0a908b720e94e3: Bootstrap replayed 1/1 log segments. Stats: ops{read=340 overwritten=0 applied=340 ignored=0} inserts{seen=1022 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:53:32.045606 7156 tablet_bootstrap.cc:492] T be42f2b9b1b747adb641149f70da608a P 158cc55ab37e4862bb0a908b720e94e3: Bootstrap complete.
I20250627 13:53:32.045940 7156 ts_tablet_manager.cc:1397] T be42f2b9b1b747adb641149f70da608a P 158cc55ab37e4862bb0a908b720e94e3: Time spent bootstrapping tablet: real 0.599s user 0.430s sys 0.039s
I20250627 13:53:32.047103 7156 raft_consensus.cc:357] T be42f2b9b1b747adb641149f70da608a P 158cc55ab37e4862bb0a908b720e94e3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false replace: true } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: true } }
I20250627 13:53:32.047317 7156 raft_consensus.cc:738] T be42f2b9b1b747adb641149f70da608a P 158cc55ab37e4862bb0a908b720e94e3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 158cc55ab37e4862bb0a908b720e94e3, State: Initialized, Role: LEARNER
I20250627 13:53:32.047595 7156 consensus_queue.cc:260] T be42f2b9b1b747adb641149f70da608a P 158cc55ab37e4862bb0a908b720e94e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 340, Last appended: 1.340, Last appended by leader: 340, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 340 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false replace: true } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: true } }
I20250627 13:53:32.048075 7156 ts_tablet_manager.cc:1428] T be42f2b9b1b747adb641149f70da608a P 158cc55ab37e4862bb0a908b720e94e3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:53:32.049022 4213 tablet_copy_service.cc:342] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Request end of tablet copy session 158cc55ab37e4862bb0a908b720e94e3-be42f2b9b1b747adb641149f70da608a received from {username='slave'} at 127.0.103.72:39721
I20250627 13:53:32.049172 4213 tablet_copy_service.cc:434] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: ending tablet copy session 158cc55ab37e4862bb0a908b720e94e3-be42f2b9b1b747adb641149f70da608a on tablet be42f2b9b1b747adb641149f70da608a with peer 158cc55ab37e4862bb0a908b720e94e3
I20250627 13:53:32.129974 7172 tablet_bootstrap.cc:492] T 1cd6f75e462546c6bb05ef14af3498fa P 158cc55ab37e4862bb0a908b720e94e3: Bootstrap replayed 1/1 log segments. Stats: ops{read=368 overwritten=0 applied=368 ignored=0} inserts{seen=1050 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:53:32.130367 7172 tablet_bootstrap.cc:492] T 1cd6f75e462546c6bb05ef14af3498fa P 158cc55ab37e4862bb0a908b720e94e3: Bootstrap complete.
I20250627 13:53:32.130645 7172 ts_tablet_manager.cc:1397] T 1cd6f75e462546c6bb05ef14af3498fa P 158cc55ab37e4862bb0a908b720e94e3: Time spent bootstrapping tablet: real 0.635s user 0.437s sys 0.043s
I20250627 13:53:32.131283 7172 raft_consensus.cc:357] T 1cd6f75e462546c6bb05ef14af3498fa P 158cc55ab37e4862bb0a908b720e94e3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: true } }
I20250627 13:53:32.131472 7172 raft_consensus.cc:738] T 1cd6f75e462546c6bb05ef14af3498fa P 158cc55ab37e4862bb0a908b720e94e3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 158cc55ab37e4862bb0a908b720e94e3, State: Initialized, Role: LEARNER
I20250627 13:53:32.131671 7172 consensus_queue.cc:260] T 1cd6f75e462546c6bb05ef14af3498fa P 158cc55ab37e4862bb0a908b720e94e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 368, Last appended: 1.368, Last appended by leader: 368, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 368 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: true } }
I20250627 13:53:32.132146 7172 ts_tablet_manager.cc:1428] T 1cd6f75e462546c6bb05ef14af3498fa P 158cc55ab37e4862bb0a908b720e94e3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250627 13:53:32.133133 4213 tablet_copy_service.cc:342] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Request end of tablet copy session 158cc55ab37e4862bb0a908b720e94e3-1cd6f75e462546c6bb05ef14af3498fa received from {username='slave'} at 127.0.103.72:39721
I20250627 13:53:32.133308 4213 tablet_copy_service.cc:434] P fb586b8cb7ab44e1a7ec5016dbd2c0a6: ending tablet copy session 158cc55ab37e4862bb0a908b720e94e3-1cd6f75e462546c6bb05ef14af3498fa on tablet 1cd6f75e462546c6bb05ef14af3498fa with peer 158cc55ab37e4862bb0a908b720e94e3
I20250627 13:53:32.200802 7186 tablet_bootstrap.cc:492] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3: Bootstrap replayed 1/1 log segments. Stats: ops{read=382 overwritten=0 applied=382 ignored=0} inserts{seen=1010 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:53:32.201292 7186 tablet_bootstrap.cc:492] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3: Bootstrap complete.
I20250627 13:53:32.201589 7186 ts_tablet_manager.cc:1397] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3: Time spent bootstrapping tablet: real 0.573s user 0.394s sys 0.053s
I20250627 13:53:32.202412 7186 raft_consensus.cc:357] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: true } }
I20250627 13:53:32.202615 7186 raft_consensus.cc:738] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 158cc55ab37e4862bb0a908b720e94e3, State: Initialized, Role: LEARNER
I20250627 13:53:32.202867 7186 consensus_queue.cc:260] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 382, Last appended: 1.382, Last appended by leader: 382, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: true } }
I20250627 13:53:32.203374 7186 ts_tablet_manager.cc:1428] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:53:32.204185 4084 tablet_copy_service.cc:342] P ad800aa8621b4fc5938692202990b5ab: Request end of tablet copy session 158cc55ab37e4862bb0a908b720e94e3-37f6cffc2bc5456e94999b60340dfeb6 received from {username='slave'} at 127.0.103.72:45649
I20250627 13:53:32.204308 4084 tablet_copy_service.cc:434] P ad800aa8621b4fc5938692202990b5ab: ending tablet copy session 158cc55ab37e4862bb0a908b720e94e3-37f6cffc2bc5456e94999b60340dfeb6 on tablet 37f6cffc2bc5456e94999b60340dfeb6 with peer 158cc55ab37e4862bb0a908b720e94e3
I20250627 13:53:32.363411 6943 raft_consensus.cc:1215] T be42f2b9b1b747adb641149f70da608a P 158cc55ab37e4862bb0a908b720e94e3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.339->[1.340-1.340] Dedup: 1.340->[]
I20250627 13:53:32.363605 6942 raft_consensus.cc:1215] T a01527b9f7c14748837f5ad4dbaa9e9d P 158cc55ab37e4862bb0a908b720e94e3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.318->[1.319-1.319] Dedup: 1.319->[]
I20250627 13:53:32.523701 6943 raft_consensus.cc:1215] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.381->[1.382-1.382] Dedup: 1.382->[]
I20250627 13:53:32.667407 6943 raft_consensus.cc:1215] T 1cd6f75e462546c6bb05ef14af3498fa P 158cc55ab37e4862bb0a908b720e94e3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.367->[1.368-1.368] Dedup: 1.368->[]
I20250627 13:53:32.792251 7085 raft_consensus.cc:1062] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6: attempting to promote NON_VOTER 158cc55ab37e4862bb0a908b720e94e3 to VOTER
I20250627 13:53:32.793042 7085 consensus_queue.cc:237] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 340, Committed index: 340, Last appended: 1.340, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false replace: true } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } }
I20250627 13:53:32.795068 5632 raft_consensus.cc:1273] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154 [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 340. Preceding OpId from leader: term: 1 index: 341. (index mismatch)
I20250627 13:53:32.795212 6943 raft_consensus.cc:1273] T be42f2b9b1b747adb641149f70da608a P 158cc55ab37e4862bb0a908b720e94e3 [term 1 LEARNER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 340. Preceding OpId from leader: term: 1 index: 341. (index mismatch)
I20250627 13:53:32.795646 7209 consensus_queue.cc:1035] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 341, Last known committed idx: 340, Time since last communication: 0.000s
I20250627 13:53:32.795984 7208 consensus_queue.cc:1035] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 341, Last known committed idx: 340, Time since last communication: 0.000s
I20250627 13:53:32.798339 7209 raft_consensus.cc:2953] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.341: config changed from index 340 to 341, 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false replace: true } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } } }
I20250627 13:53:32.799039 5632 raft_consensus.cc:2953] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154 [term 1 FOLLOWER]: Committing config change with OpId 1.341: config changed from index 340 to 341, 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false replace: true } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } } }
I20250627 13:53:32.800084 6943 raft_consensus.cc:2953] T be42f2b9b1b747adb641149f70da608a P 158cc55ab37e4862bb0a908b720e94e3 [term 1 FOLLOWER]: Committing config change with OpId 1.341: config changed from index 340 to 341, 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false replace: true } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } } }
I20250627 13:53:32.803332 3790 catalog_manager.cc:5582] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 reported cstate change: config changed from index 340 to 341, 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 341 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6503b59977444aaebe5ee70a0bc99154" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 45413 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:53:32.806191 7108 raft_consensus.cc:1062] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab: attempting to promote NON_VOTER 158cc55ab37e4862bb0a908b720e94e3 to VOTER
I20250627 13:53:32.806811 7108 consensus_queue.cc:237] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 319, Committed index: 319, Last appended: 1.319, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } }
I20250627 13:53:32.808877 3930 raft_consensus.cc:1273] T a01527b9f7c14748837f5ad4dbaa9e9d P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Refusing update from remote peer ad800aa8621b4fc5938692202990b5ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 319. Preceding OpId from leader: term: 1 index: 320. (index mismatch)
I20250627 13:53:32.808905 6943 raft_consensus.cc:1273] T a01527b9f7c14748837f5ad4dbaa9e9d P 158cc55ab37e4862bb0a908b720e94e3 [term 1 LEARNER]: Refusing update from remote peer ad800aa8621b4fc5938692202990b5ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 319. Preceding OpId from leader: term: 1 index: 320. (index mismatch)
I20250627 13:53:32.809360 7200 consensus_queue.cc:1035] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 320, Last known committed idx: 319, Time since last communication: 0.000s
I20250627 13:53:32.809605 7196 consensus_queue.cc:1035] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 320, Last known committed idx: 319, Time since last communication: 0.000s
I20250627 13:53:32.812072 7196 raft_consensus.cc:2953] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab [term 1 LEADER]: Committing config change with OpId 1.320: config changed from index 319 to 320, 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } } }
I20250627 13:53:32.814320 6943 raft_consensus.cc:2953] T a01527b9f7c14748837f5ad4dbaa9e9d P 158cc55ab37e4862bb0a908b720e94e3 [term 1 FOLLOWER]: Committing config change with OpId 1.320: config changed from index 319 to 320, 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } } }
I20250627 13:53:32.815892 3930 raft_consensus.cc:2953] T a01527b9f7c14748837f5ad4dbaa9e9d P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Committing config change with OpId 1.320: config changed from index 319 to 320, 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } } }
I20250627 13:53:32.816320 3784 catalog_manager.cc:5582] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab reported cstate change: config changed from index 319 to 320, 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ad800aa8621b4fc5938692202990b5ab" committed_config { opid_index: 320 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:53:32.822100 4194 consensus_queue.cc:237] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 341, Committed index: 341, Last appended: 1.341, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } }
I20250627 13:53:32.823915 6943 raft_consensus.cc:1273] T be42f2b9b1b747adb641149f70da608a P 158cc55ab37e4862bb0a908b720e94e3 [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 341. Preceding OpId from leader: term: 1 index: 342. (index mismatch)
I20250627 13:53:32.824013 4064 consensus_queue.cc:237] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 320, Committed index: 320, Last appended: 1.320, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } }
I20250627 13:53:32.824649 7085 consensus_queue.cc:1035] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 342, Last known committed idx: 341, Time since last communication: 0.000s
I20250627 13:53:32.825615 6942 raft_consensus.cc:1273] T a01527b9f7c14748837f5ad4dbaa9e9d P 158cc55ab37e4862bb0a908b720e94e3 [term 1 FOLLOWER]: Refusing update from remote peer ad800aa8621b4fc5938692202990b5ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 320. Preceding OpId from leader: term: 1 index: 321. (index mismatch)
I20250627 13:53:32.826265 7108 consensus_queue.cc:1035] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 321, Last known committed idx: 320, Time since last communication: 0.000s
I20250627 13:53:32.826629 7208 raft_consensus.cc:2953] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.342: config changed from index 341 to 342, VOTER 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) evicted. New config: { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } } }
I20250627 13:53:32.827221 6943 raft_consensus.cc:2953] T be42f2b9b1b747adb641149f70da608a P 158cc55ab37e4862bb0a908b720e94e3 [term 1 FOLLOWER]: Committing config change with OpId 1.342: config changed from index 341 to 342, VOTER 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) evicted. New config: { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } } }
I20250627 13:53:32.828462 7108 raft_consensus.cc:2953] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab [term 1 LEADER]: Committing config change with OpId 1.321: config changed from index 320 to 321, VOTER f46fd51df5864af9b8493acd69db0747 (127.0.103.65) evicted. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } } }
I20250627 13:53:32.830794 3779 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet be42f2b9b1b747adb641149f70da608a with cas_config_opid_index 341: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:53:32.831638 3777 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet a01527b9f7c14748837f5ad4dbaa9e9d with cas_config_opid_index 320: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:53:32.832163 6943 raft_consensus.cc:2953] T a01527b9f7c14748837f5ad4dbaa9e9d P 158cc55ab37e4862bb0a908b720e94e3 [term 1 FOLLOWER]: Committing config change with OpId 1.321: config changed from index 320 to 321, VOTER f46fd51df5864af9b8493acd69db0747 (127.0.103.65) evicted. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } } }
I20250627 13:53:32.833637 3784 catalog_manager.cc:5582] T be42f2b9b1b747adb641149f70da608a P fb586b8cb7ab44e1a7ec5016dbd2c0a6 reported cstate change: config changed from index 341 to 342, VOTER 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71) evicted. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:53:32.835328 3788 catalog_manager.cc:5582] T a01527b9f7c14748837f5ad4dbaa9e9d P ad800aa8621b4fc5938692202990b5ab reported cstate change: config changed from index 320 to 321, VOTER f46fd51df5864af9b8493acd69db0747 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "ad800aa8621b4fc5938692202990b5ab" committed_config { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:53:32.837386 5614 tablet_service.cc:1515] Processing DeleteTablet for tablet be42f2b9b1b747adb641149f70da608a with delete_type TABLET_DATA_TOMBSTONED (TS 6503b59977444aaebe5ee70a0bc99154 not found in new config with opid_index 342) from {username='slave'} at 127.0.0.1:58512
I20250627 13:53:32.838587 3912 tablet_service.cc:1515] Processing DeleteTablet for tablet a01527b9f7c14748837f5ad4dbaa9e9d with delete_type TABLET_DATA_TOMBSTONED (TS f46fd51df5864af9b8493acd69db0747 not found in new config with opid_index 321) from {username='slave'} at 127.0.0.1:41436
I20250627 13:53:32.838667 7290 tablet_replica.cc:331] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154: stopping tablet replica
I20250627 13:53:32.840209 7290 raft_consensus.cc:2241] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:53:32.840566 7290 raft_consensus.cc:2270] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:53:32.841167 7291 tablet_replica.cc:331] T a01527b9f7c14748837f5ad4dbaa9e9d P f46fd51df5864af9b8493acd69db0747: stopping tablet replica
I20250627 13:53:32.841352 7291 raft_consensus.cc:2241] T a01527b9f7c14748837f5ad4dbaa9e9d P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:53:32.841606 7291 raft_consensus.cc:2270] T a01527b9f7c14748837f5ad4dbaa9e9d P f46fd51df5864af9b8493acd69db0747 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:53:32.842625 7290 ts_tablet_manager.cc:1905] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:53:32.843055 7291 ts_tablet_manager.cc:1905] T a01527b9f7c14748837f5ad4dbaa9e9d P f46fd51df5864af9b8493acd69db0747: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:53:32.846891 7290 ts_tablet_manager.cc:1918] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.341
I20250627 13:53:32.846992 7290 log.cc:1199] T be42f2b9b1b747adb641149f70da608a P 6503b59977444aaebe5ee70a0bc99154: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/wal/wals/be42f2b9b1b747adb641149f70da608a
I20250627 13:53:32.847050 7291 ts_tablet_manager.cc:1918] T a01527b9f7c14748837f5ad4dbaa9e9d P f46fd51df5864af9b8493acd69db0747: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.320
I20250627 13:53:32.847159 7291 log.cc:1199] T a01527b9f7c14748837f5ad4dbaa9e9d P f46fd51df5864af9b8493acd69db0747: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/wals/a01527b9f7c14748837f5ad4dbaa9e9d
I20250627 13:53:32.847749 3777 catalog_manager.cc:4928] TS 6503b59977444aaebe5ee70a0bc99154 (127.0.103.71:45413): tablet be42f2b9b1b747adb641149f70da608a (table rebalance_test_table_1 [id=70cec1ac0b6b4a82a338f38f22668b39]) successfully deleted
I20250627 13:53:32.847838 3776 catalog_manager.cc:4928] TS f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133): tablet a01527b9f7c14748837f5ad4dbaa9e9d (table rebalance_test_table_4 [id=b5d412773a2343d1919352385a3629db]) successfully deleted
I20250627 13:53:32.967732 7200 raft_consensus.cc:1062] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab: attempting to promote NON_VOTER 158cc55ab37e4862bb0a908b720e94e3 to VOTER
I20250627 13:53:32.968550 7200 consensus_queue.cc:237] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 382, Committed index: 382, Last appended: 1.382, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 383 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } }
I20250627 13:53:32.971011 6943 raft_consensus.cc:1273] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3 [term 1 LEARNER]: Refusing update from remote peer ad800aa8621b4fc5938692202990b5ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 382. Preceding OpId from leader: term: 1 index: 383. (index mismatch)
I20250627 13:53:32.971071 5369 raft_consensus.cc:1273] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1 [term 1 FOLLOWER]: Refusing update from remote peer ad800aa8621b4fc5938692202990b5ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 382. Preceding OpId from leader: term: 1 index: 383. (index mismatch)
I20250627 13:53:32.971724 7196 consensus_queue.cc:1035] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 383, Last known committed idx: 382, Time since last communication: 0.000s
I20250627 13:53:32.972352 7108 consensus_queue.cc:1035] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 383, Last known committed idx: 382, Time since last communication: 0.000s
I20250627 13:53:32.974752 7108 raft_consensus.cc:2953] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [term 1 LEADER]: Committing config change with OpId 1.383: config changed from index 382 to 383, 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 383 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } } }
I20250627 13:53:32.975790 6943 raft_consensus.cc:2953] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3 [term 1 FOLLOWER]: Committing config change with OpId 1.383: config changed from index 382 to 383, 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 383 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } } }
I20250627 13:53:32.977883 5369 raft_consensus.cc:2953] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1 [term 1 FOLLOWER]: Committing config change with OpId 1.383: config changed from index 382 to 383, 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 383 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } } }
I20250627 13:53:32.980326 3783 catalog_manager.cc:5582] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3 reported cstate change: config changed from index 382 to 383, 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ad800aa8621b4fc5938692202990b5ab" committed_config { opid_index: 383 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } } }
I20250627 13:53:32.992810 4064 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "37f6cffc2bc5456e94999b60340dfeb6"
dest_uuid: "ad800aa8621b4fc5938692202990b5ab"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47040
I20250627 13:53:32.993003 4064 raft_consensus.cc:604] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [term 1 LEADER]: Received request to transfer leadership
I20250627 13:53:33.220646 4064 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "37f6cffc2bc5456e94999b60340dfeb6"
dest_uuid: "ad800aa8621b4fc5938692202990b5ab"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47062
I20250627 13:53:33.220839 4064 raft_consensus.cc:604] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [term 1 LEADER]: Received request to transfer leadership
I20250627 13:53:33.233245 7208 raft_consensus.cc:1062] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6: attempting to promote NON_VOTER 158cc55ab37e4862bb0a908b720e94e3 to VOTER
I20250627 13:53:33.233902 7208 consensus_queue.cc:237] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 368, Committed index: 368, Last appended: 1.368, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } }
I20250627 13:53:33.236204 5369 raft_consensus.cc:1273] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1 [term 1 FOLLOWER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 368. Preceding OpId from leader: term: 1 index: 369. (index mismatch)
I20250627 13:53:33.236301 6943 raft_consensus.cc:1273] T 1cd6f75e462546c6bb05ef14af3498fa P 158cc55ab37e4862bb0a908b720e94e3 [term 1 LEARNER]: Refusing update from remote peer fb586b8cb7ab44e1a7ec5016dbd2c0a6: Log matching property violated. Preceding OpId in replica: term: 1 index: 368. Preceding OpId from leader: term: 1 index: 369. (index mismatch)
I20250627 13:53:33.236804 7209 consensus_queue.cc:1035] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 369, Last known committed idx: 368, Time since last communication: 0.000s
I20250627 13:53:33.237238 7209 consensus_queue.cc:1035] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 369, Last known committed idx: 368, Time since last communication: 0.000s
I20250627 13:53:33.239374 7209 raft_consensus.cc:2953] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Committing config change with OpId 1.369: config changed from index 368 to 369, 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } } }
I20250627 13:53:33.240235 6943 raft_consensus.cc:2953] T 1cd6f75e462546c6bb05ef14af3498fa P 158cc55ab37e4862bb0a908b720e94e3 [term 1 FOLLOWER]: Committing config change with OpId 1.369: config changed from index 368 to 369, 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } } }
I20250627 13:53:33.240314 5369 raft_consensus.cc:2953] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1 [term 1 FOLLOWER]: Committing config change with OpId 1.369: config changed from index 368 to 369, 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } } }
I20250627 13:53:33.244336 3788 catalog_manager.cc:5582] T 1cd6f75e462546c6bb05ef14af3498fa P 158cc55ab37e4862bb0a908b720e94e3 reported cstate change: config changed from index 368 to 369, 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" committed_config { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } } }
I20250627 13:53:33.440661 7200 raft_consensus.cc:991] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab: : Instructing follower 158cc55ab37e4862bb0a908b720e94e3 to start an election
I20250627 13:53:33.440865 7200 raft_consensus.cc:1079] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [term 1 LEADER]: Signalling peer 158cc55ab37e4862bb0a908b720e94e3 to start an election
I20250627 13:53:33.441687 6943 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "37f6cffc2bc5456e94999b60340dfeb6"
dest_uuid: "158cc55ab37e4862bb0a908b720e94e3"
from {username='slave'} at 127.0.103.66:46911
I20250627 13:53:33.441954 6943 raft_consensus.cc:491] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:53:33.442107 6943 raft_consensus.cc:3058] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:53:33.444412 6943 raft_consensus.cc:513] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 383 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } }
I20250627 13:53:33.446094 6943 leader_election.cc:290] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3 [CANDIDATE]: Term 2 election: Requested vote from peers ad800aa8621b4fc5938692202990b5ab (127.0.103.66:43273), 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69:42665)
I20250627 13:53:33.446427 4064 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37f6cffc2bc5456e94999b60340dfeb6" candidate_uuid: "158cc55ab37e4862bb0a908b720e94e3" candidate_term: 2 candidate_status { last_received { term: 1 index: 383 } } ignore_live_leader: true dest_uuid: "ad800aa8621b4fc5938692202990b5ab"
I20250627 13:53:33.446642 4064 raft_consensus.cc:3053] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [term 1 LEADER]: Stepping down as leader of term 1
I20250627 13:53:33.447108 4064 raft_consensus.cc:738] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ad800aa8621b4fc5938692202990b5ab, State: Running, Role: LEADER
I20250627 13:53:33.447445 4064 consensus_queue.cc:260] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 383, Committed index: 383, Last appended: 1.383, Last appended by leader: 383, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 383 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } }
I20250627 13:53:33.447896 4064 raft_consensus.cc:3058] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:53:33.449606 4062 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "37f6cffc2bc5456e94999b60340dfeb6"
dest_uuid: "ad800aa8621b4fc5938692202990b5ab"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47086
I20250627 13:53:33.450766 4064 raft_consensus.cc:2466] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158cc55ab37e4862bb0a908b720e94e3 in term 2.
I20250627 13:53:33.450912 4062 raft_consensus.cc:604] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [term 2 FOLLOWER]: Received request to transfer leadership
I20250627 13:53:33.451025 4062 raft_consensus.cc:612] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250627 13:53:33.451305 6877 leader_election.cc:304] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158cc55ab37e4862bb0a908b720e94e3, ad800aa8621b4fc5938692202990b5ab; no voters:
I20250627 13:53:33.451673 7211 raft_consensus.cc:2802] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3 [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:53:33.452203 7211 raft_consensus.cc:695] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3 [term 2 LEADER]: Becoming Leader. State: Replica: 158cc55ab37e4862bb0a908b720e94e3, State: Running, Role: LEADER
I20250627 13:53:33.452721 7211 consensus_queue.cc:237] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 383, Committed index: 383, Last appended: 1.383, Last appended by leader: 383, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 383 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } }
I20250627 13:53:33.456095 5369 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37f6cffc2bc5456e94999b60340dfeb6" candidate_uuid: "158cc55ab37e4862bb0a908b720e94e3" candidate_term: 2 candidate_status { last_received { term: 1 index: 383 } } ignore_live_leader: true dest_uuid: "911714a383074e09a9b2d127ab46d2c1"
I20250627 13:53:33.456317 5369 raft_consensus.cc:3058] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:53:33.456693 3783 catalog_manager.cc:5582] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3 reported cstate change: term changed from 1 to 2, leader changed from ad800aa8621b4fc5938692202990b5ab (127.0.103.66) to 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72). New cstate: current_term: 2 leader_uuid: "158cc55ab37e4862bb0a908b720e94e3" committed_config { opid_index: 383 OBSOLETE_local: false peers { permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:53:33.458433 5369 raft_consensus.cc:2466] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 158cc55ab37e4862bb0a908b720e94e3 in term 2.
I20250627 13:53:33.474278 4194 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "1cd6f75e462546c6bb05ef14af3498fa"
dest_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40036
I20250627 13:53:33.474458 4194 raft_consensus.cc:604] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Received request to transfer leadership
I20250627 13:53:33.700516 4194 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "1cd6f75e462546c6bb05ef14af3498fa"
dest_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40062
I20250627 13:53:33.700701 4194 raft_consensus.cc:604] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Received request to transfer leadership
I20250627 13:53:33.772616 7389 raft_consensus.cc:991] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6: : Instructing follower 911714a383074e09a9b2d127ab46d2c1 to start an election
I20250627 13:53:33.772774 7389 raft_consensus.cc:1079] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Signalling peer 911714a383074e09a9b2d127ab46d2c1 to start an election
I20250627 13:53:33.773398 5369 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "1cd6f75e462546c6bb05ef14af3498fa"
dest_uuid: "911714a383074e09a9b2d127ab46d2c1"
from {username='slave'} at 127.0.103.67:52885
I20250627 13:53:33.773617 5369 raft_consensus.cc:491] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:53:33.773747 5369 raft_consensus.cc:3058] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:53:33.775674 5369 raft_consensus.cc:513] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } }
I20250627 13:53:33.776288 5369 leader_election.cc:290] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1 [CANDIDATE]: Term 2 election: Requested vote from peers fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67:44627), 158cc55ab37e4862bb0a908b720e94e3 (127.0.103.72:45709)
I20250627 13:53:33.776719 4194 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1cd6f75e462546c6bb05ef14af3498fa" candidate_uuid: "911714a383074e09a9b2d127ab46d2c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 369 } } ignore_live_leader: true dest_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6"
I20250627 13:53:33.776922 4194 raft_consensus.cc:3053] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Stepping down as leader of term 1
I20250627 13:53:33.777030 4194 raft_consensus.cc:738] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb586b8cb7ab44e1a7ec5016dbd2c0a6, State: Running, Role: LEADER
I20250627 13:53:33.777289 4194 consensus_queue.cc:260] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 369, Committed index: 369, Last appended: 1.369, Last appended by leader: 369, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } }
I20250627 13:53:33.777678 4194 raft_consensus.cc:3058] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:53:33.780076 4194 raft_consensus.cc:2466] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 911714a383074e09a9b2d127ab46d2c1 in term 2.
I20250627 13:53:33.780519 5308 leader_election.cc:304] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 911714a383074e09a9b2d127ab46d2c1, fb586b8cb7ab44e1a7ec5016dbd2c0a6; no voters:
I20250627 13:53:33.780828 7258 raft_consensus.cc:2802] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1 [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:53:33.781347 7258 raft_consensus.cc:695] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1 [term 2 LEADER]: Becoming Leader. State: Replica: 911714a383074e09a9b2d127ab46d2c1, State: Running, Role: LEADER
I20250627 13:53:33.781776 7258 consensus_queue.cc:237] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 369, Committed index: 369, Last appended: 1.369, Last appended by leader: 369, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } }
I20250627 13:53:33.783399 6943 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1cd6f75e462546c6bb05ef14af3498fa" candidate_uuid: "911714a383074e09a9b2d127ab46d2c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 369 } } ignore_live_leader: true dest_uuid: "158cc55ab37e4862bb0a908b720e94e3"
I20250627 13:53:33.783653 6943 raft_consensus.cc:3058] T 1cd6f75e462546c6bb05ef14af3498fa P 158cc55ab37e4862bb0a908b720e94e3 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:53:33.785549 6943 raft_consensus.cc:2466] T 1cd6f75e462546c6bb05ef14af3498fa P 158cc55ab37e4862bb0a908b720e94e3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 911714a383074e09a9b2d127ab46d2c1 in term 2.
I20250627 13:53:33.786604 3783 catalog_manager.cc:5582] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1 reported cstate change: term changed from 1 to 2, leader changed from fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67) to 911714a383074e09a9b2d127ab46d2c1 (127.0.103.69). New cstate: current_term: 2 leader_uuid: "911714a383074e09a9b2d127ab46d2c1" committed_config { opid_index: 369 OBSOLETE_local: false peers { permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:33.993691 4062 raft_consensus.cc:1273] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [term 2 FOLLOWER]: Refusing update from remote peer 158cc55ab37e4862bb0a908b720e94e3: Log matching property violated. Preceding OpId in replica: term: 1 index: 383. Preceding OpId from leader: term: 2 index: 384. (index mismatch)
I20250627 13:53:33.994256 7402 consensus_queue.cc:1035] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad800aa8621b4fc5938692202990b5ab" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43273 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 384, Last known committed idx: 383, Time since last communication: 0.000s
I20250627 13:53:33.998366 5369 raft_consensus.cc:1273] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1 [term 2 FOLLOWER]: Refusing update from remote peer 158cc55ab37e4862bb0a908b720e94e3: Log matching property violated. Preceding OpId in replica: term: 1 index: 383. Preceding OpId from leader: term: 2 index: 384. (index mismatch)
I20250627 13:53:33.998920 7402 consensus_queue.cc:1035] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 384, Last known committed idx: 383, Time since last communication: 0.000s
I20250627 13:53:34.331032 4194 raft_consensus.cc:1273] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 2 FOLLOWER]: Refusing update from remote peer 911714a383074e09a9b2d127ab46d2c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 369. Preceding OpId from leader: term: 2 index: 370. (index mismatch)
I20250627 13:53:34.331593 7258 consensus_queue.cc:1035] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb586b8cb7ab44e1a7ec5016dbd2c0a6" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44627 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 370, Last known committed idx: 369, Time since last communication: 0.000s
I20250627 13:53:34.335613 6942 raft_consensus.cc:1273] T 1cd6f75e462546c6bb05ef14af3498fa P 158cc55ab37e4862bb0a908b720e94e3 [term 2 FOLLOWER]: Refusing update from remote peer 911714a383074e09a9b2d127ab46d2c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 369. Preceding OpId from leader: term: 2 index: 370. (index mismatch)
I20250627 13:53:34.336159 7421 consensus_queue.cc:1035] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 370, Last known committed idx: 369, Time since last communication: 0.000s
I20250627 13:53:34.394914 6942 consensus_queue.cc:237] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 384, Committed index: 384, Last appended: 2.384, Last appended by leader: 383, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } }
I20250627 13:53:34.396530 5369 raft_consensus.cc:1273] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1 [term 2 FOLLOWER]: Refusing update from remote peer 158cc55ab37e4862bb0a908b720e94e3: Log matching property violated. Preceding OpId in replica: term: 2 index: 384. Preceding OpId from leader: term: 2 index: 385. (index mismatch)
I20250627 13:53:34.397102 7435 consensus_queue.cc:1035] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 385, Last known committed idx: 384, Time since last communication: 0.000s
I20250627 13:53:34.399205 7404 raft_consensus.cc:2953] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3 [term 2 LEADER]: Committing config change with OpId 2.385: config changed from index 383 to 385, VOTER ad800aa8621b4fc5938692202990b5ab (127.0.103.66) evicted. New config: { opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } } }
I20250627 13:53:34.399799 5369 raft_consensus.cc:2953] T 37f6cffc2bc5456e94999b60340dfeb6 P 911714a383074e09a9b2d127ab46d2c1 [term 2 FOLLOWER]: Committing config change with OpId 2.385: config changed from index 383 to 385, VOTER ad800aa8621b4fc5938692202990b5ab (127.0.103.66) evicted. New config: { opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } } }
I20250627 13:53:34.401973 3777 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 37f6cffc2bc5456e94999b60340dfeb6 with cas_config_opid_index 383: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:53:34.403256 3784 catalog_manager.cc:5582] T 37f6cffc2bc5456e94999b60340dfeb6 P 158cc55ab37e4862bb0a908b720e94e3 reported cstate change: config changed from index 383 to 385, VOTER ad800aa8621b4fc5938692202990b5ab (127.0.103.66) evicted. New cstate: current_term: 2 leader_uuid: "158cc55ab37e4862bb0a908b720e94e3" committed_config { opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:53:34.407212 4042 tablet_service.cc:1515] Processing DeleteTablet for tablet 37f6cffc2bc5456e94999b60340dfeb6 with delete_type TABLET_DATA_TOMBSTONED (TS ad800aa8621b4fc5938692202990b5ab not found in new config with opid_index 385) from {username='slave'} at 127.0.0.1:33136
I20250627 13:53:34.407850 7440 tablet_replica.cc:331] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab: stopping tablet replica
I20250627 13:53:34.408088 7440 raft_consensus.cc:2241] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:53:34.408272 7440 raft_consensus.cc:2270] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:53:34.409816 7440 ts_tablet_manager.cc:1905] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:53:34.414103 7440 ts_tablet_manager.cc:1918] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.384
I20250627 13:53:34.414242 7440 log.cc:1199] T 37f6cffc2bc5456e94999b60340dfeb6 P ad800aa8621b4fc5938692202990b5ab: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/wals/37f6cffc2bc5456e94999b60340dfeb6
I20250627 13:53:34.414997 3777 catalog_manager.cc:4928] TS ad800aa8621b4fc5938692202990b5ab (127.0.103.66:43273): tablet 37f6cffc2bc5456e94999b60340dfeb6 (table rebalance_test_table_2 [id=41b84e234b64410a9ea4c057e1c398cd]) successfully deleted
I20250627 13:53:34.929490 5369 consensus_queue.cc:237] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 370, Committed index: 370, Last appended: 2.370, Last appended by leader: 369, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } }
I20250627 13:53:34.931051 6942 raft_consensus.cc:1273] T 1cd6f75e462546c6bb05ef14af3498fa P 158cc55ab37e4862bb0a908b720e94e3 [term 2 FOLLOWER]: Refusing update from remote peer 911714a383074e09a9b2d127ab46d2c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 370. Preceding OpId from leader: term: 2 index: 371. (index mismatch)
I20250627 13:53:34.931525 7464 consensus_queue.cc:1035] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 371, Last known committed idx: 370, Time since last communication: 0.000s
I20250627 13:53:34.933429 7463 raft_consensus.cc:2953] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1 [term 2 LEADER]: Committing config change with OpId 2.371: config changed from index 369 to 371, VOTER fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67) evicted. New config: { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } } }
I20250627 13:53:34.933988 6942 raft_consensus.cc:2953] T 1cd6f75e462546c6bb05ef14af3498fa P 158cc55ab37e4862bb0a908b720e94e3 [term 2 FOLLOWER]: Committing config change with OpId 2.371: config changed from index 369 to 371, VOTER fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67) evicted. New config: { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } } }
I20250627 13:53:34.936007 3777 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 1cd6f75e462546c6bb05ef14af3498fa with cas_config_opid_index 369: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:53:34.937623 3784 catalog_manager.cc:5582] T 1cd6f75e462546c6bb05ef14af3498fa P 911714a383074e09a9b2d127ab46d2c1 reported cstate change: config changed from index 369 to 371, VOTER fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67) evicted. New cstate: current_term: 2 leader_uuid: "911714a383074e09a9b2d127ab46d2c1" committed_config { opid_index: 371 OBSOLETE_local: false peers { permanent_uuid: "911714a383074e09a9b2d127ab46d2c1" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 42665 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "158cc55ab37e4862bb0a908b720e94e3" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 45709 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:53:34.941375 4174 tablet_service.cc:1515] Processing DeleteTablet for tablet 1cd6f75e462546c6bb05ef14af3498fa with delete_type TABLET_DATA_TOMBSTONED (TS fb586b8cb7ab44e1a7ec5016dbd2c0a6 not found in new config with opid_index 371) from {username='slave'} at 127.0.0.1:32882
I20250627 13:53:34.942015 7469 tablet_replica.cc:331] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6: stopping tablet replica
I20250627 13:53:34.942215 7469 raft_consensus.cc:2241] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:53:34.942423 7469 raft_consensus.cc:2270] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:53:34.944077 7469 ts_tablet_manager.cc:1905] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:53:34.948987 7469 ts_tablet_manager.cc:1918] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.370
I20250627 13:53:34.949105 7469 log.cc:1199] T 1cd6f75e462546c6bb05ef14af3498fa P fb586b8cb7ab44e1a7ec5016dbd2c0a6: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal/wals/1cd6f75e462546c6bb05ef14af3498fa
I20250627 13:53:34.949898 3779 catalog_manager.cc:4928] TS fb586b8cb7ab44e1a7ec5016dbd2c0a6 (127.0.103.67:44627): tablet 1cd6f75e462546c6bb05ef14af3498fa (table rebalance_test_table_3 [id=d219e402e8284b468ef8ca703004dd5f]) successfully deleted
I20250627 13:53:35.131809 5221 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:35.133041 6923 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:35.133595 5614 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:35.137231 5352 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
I20250627 13:53:35.155843 3912 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
I20250627 13:53:35.163232 4174 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:53:35.174062 5482 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
I20250627 13:53:35.175899 4042 tablet_service.cc:1430] Tablet server has 3 leaders and 0 scanners
I20250627 13:53:35.419699 6923 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:35.422093 5221 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:35.441160 5482 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
I20250627 13:53:35.453095 5352 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
I20250627 13:53:35.453799 5614 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:35.466473 4042 tablet_service.cc:1430] Tablet server has 3 leaders and 0 scanners
I20250627 13:53:35.471608 3912 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
I20250627 13:53:35.503044 4174 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:53:35.797127 5221 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:35.809413 6923 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:35.810122 5482 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
I20250627 13:53:35.821390 3912 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
I20250627 13:53:35.827127 4174 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:53:35.848534 5614 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:53:35.848764 4042 tablet_service.cc:1430] Tablet server has 3 leaders and 0 scanners
I20250627 13:53:35.849311 5352 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bcc04dadc3364622b3a0e2d593b2dc60 | 127.0.103.126:39869 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.103.84:41153 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
158cc55ab37e4862bb0a908b720e94e3 | 127.0.103.72:45709 | HEALTHY | <none> | 1 | 0
36100389495d4971b10cf590bc82c42c | 127.0.103.68:45503 | HEALTHY | <none> | 1 | 0
6503b59977444aaebe5ee70a0bc99154 | 127.0.103.71:45413 | HEALTHY | <none> | 1 | 0
911714a383074e09a9b2d127ab46d2c1 | 127.0.103.69:42665 | HEALTHY | <none> | 2 | 0
ad800aa8621b4fc5938692202990b5ab | 127.0.103.66:43273 | HEALTHY | <none> | 3 | 0
bf0be937c3634bcab62f1e58402a776f | 127.0.103.70:37215 | HEALTHY | <none> | 2 | 0
f46fd51df5864af9b8493acd69db0747 | 127.0.103.65:35133 | HEALTHY | <none> | 2 | 0
fb586b8cb7ab44e1a7ec5016dbd2c0a6 | 127.0.103.67:44627 | HEALTHY | <none> | 4 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.0.103.65 | experimental | 127.0.103.65:35133
local_ip_for_outbound_sockets | 127.0.103.66 | experimental | 127.0.103.66:43273
local_ip_for_outbound_sockets | 127.0.103.67 | experimental | 127.0.103.67:44627
local_ip_for_outbound_sockets | 127.0.103.68 | experimental | 127.0.103.68:45503
local_ip_for_outbound_sockets | 127.0.103.69 | experimental | 127.0.103.69:42665
local_ip_for_outbound_sockets | 127.0.103.70 | experimental | 127.0.103.70:37215
local_ip_for_outbound_sockets | 127.0.103.71 | experimental | 127.0.103.71:45413
local_ip_for_outbound_sockets | 127.0.103.72 | experimental | 127.0.103.72:45709
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.103.65:35133
server_dump_info_path | /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.103.66:43273
server_dump_info_path | /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.103.67:44627
server_dump_info_path | /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.103.68:45503
server_dump_info_path | /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.103.69:42665
server_dump_info_path | /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.103.70:37215
server_dump_info_path | /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.103.71:45413
server_dump_info_path | /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.0.103.72:45709
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.103.84:41153 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250627 13:53:35.922451 413 log_verifier.cc:126] Checking tablet 0fdcc98da13c4cf998b1fd1b3e3f9fe5
I20250627 13:53:35.933272 413 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 0fdcc98da13c4cf998b1fd1b3e3f9fe5
I20250627 13:53:35.933378 413 log_verifier.cc:126] Checking tablet 16fed4ebf25c44069b67af98964ab075
I20250627 13:53:36.054546 413 log_verifier.cc:177] Verified matching terms for 337 ops in tablet 16fed4ebf25c44069b67af98964ab075
I20250627 13:53:36.054922 413 log_verifier.cc:126] Checking tablet 1cd6f75e462546c6bb05ef14af3498fa
I20250627 13:53:36.190026 413 log_verifier.cc:177] Verified matching terms for 371 ops in tablet 1cd6f75e462546c6bb05ef14af3498fa
I20250627 13:53:36.190445 413 log_verifier.cc:126] Checking tablet 37f6cffc2bc5456e94999b60340dfeb6
I20250627 13:53:36.340026 413 log_verifier.cc:177] Verified matching terms for 385 ops in tablet 37f6cffc2bc5456e94999b60340dfeb6
I20250627 13:53:36.340483 413 log_verifier.cc:126] Checking tablet 3e2b11c6788e4ccdbdde60769fee1aae
I20250627 13:53:36.462354 413 log_verifier.cc:177] Verified matching terms for 348 ops in tablet 3e2b11c6788e4ccdbdde60769fee1aae
I20250627 13:53:36.462800 413 log_verifier.cc:126] Checking tablet 40b04be88e9947c5a5ca8432e3791bdd
I20250627 13:53:36.599139 413 log_verifier.cc:177] Verified matching terms for 363 ops in tablet 40b04be88e9947c5a5ca8432e3791bdd
I20250627 13:53:36.599771 413 log_verifier.cc:126] Checking tablet 4a118f20daaf45a2a1c539410dbf0d30
I20250627 13:53:36.742538 413 log_verifier.cc:177] Verified matching terms for 392 ops in tablet 4a118f20daaf45a2a1c539410dbf0d30
I20250627 13:53:36.743185 413 log_verifier.cc:126] Checking tablet 76518bc59dfe443abd8e97539d1bcb8a
I20250627 13:53:36.877620 413 log_verifier.cc:177] Verified matching terms for 385 ops in tablet 76518bc59dfe443abd8e97539d1bcb8a
I20250627 13:53:36.878137 413 log_verifier.cc:126] Checking tablet 8fe13ad746124b1d82cfb8ce691828a4
I20250627 13:53:36.994144 413 log_verifier.cc:177] Verified matching terms for 325 ops in tablet 8fe13ad746124b1d82cfb8ce691828a4
I20250627 13:53:36.994563 413 log_verifier.cc:126] Checking tablet 978e20547d644d3988d6da29235903aa
I20250627 13:53:37.106506 413 log_verifier.cc:177] Verified matching terms for 317 ops in tablet 978e20547d644d3988d6da29235903aa
I20250627 13:53:37.106920 413 log_verifier.cc:126] Checking tablet 9f374c0cbcd0493597c8f30e3e63d962
I20250627 13:53:37.254024 413 log_verifier.cc:177] Verified matching terms for 394 ops in tablet 9f374c0cbcd0493597c8f30e3e63d962
I20250627 13:53:37.254488 413 log_verifier.cc:126] Checking tablet a01527b9f7c14748837f5ad4dbaa9e9d
I20250627 13:53:37.377012 413 log_verifier.cc:177] Verified matching terms for 321 ops in tablet a01527b9f7c14748837f5ad4dbaa9e9d
I20250627 13:53:37.377439 413 log_verifier.cc:126] Checking tablet b2b66876e09748b49f27128fe7591be9
I20250627 13:53:37.499989 413 log_verifier.cc:177] Verified matching terms for 348 ops in tablet b2b66876e09748b49f27128fe7591be9
I20250627 13:53:37.500551 413 log_verifier.cc:126] Checking tablet be42f2b9b1b747adb641149f70da608a
I20250627 13:53:37.635279 413 log_verifier.cc:177] Verified matching terms for 342 ops in tablet be42f2b9b1b747adb641149f70da608a
I20250627 13:53:37.635710 413 log_verifier.cc:126] Checking tablet c20692dbde954a259acdb9c765978113
I20250627 13:53:37.754618 413 log_verifier.cc:177] Verified matching terms for 329 ops in tablet c20692dbde954a259acdb9c765978113
I20250627 13:53:37.755029 413 log_verifier.cc:126] Checking tablet e79731c0edcc4c11a573cf46f0b2c06a
I20250627 13:53:37.877442 413 log_verifier.cc:177] Verified matching terms for 341 ops in tablet e79731c0edcc4c11a573cf46f0b2c06a
I20250627 13:53:37.886931 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 3851
I20250627 13:53:37.888098 3972 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:53:38.095726 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 3851
W20250627 13:53:38.131510 5435 connection.cc:537] client connection to 127.0.103.65:35133 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250627 13:53:38.131773 3998 connection.cc:537] server connection from 127.0.103.65:60079 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250627 13:53:38.131811 5435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250627 13:53:38.132226 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 3982
W20250627 13:53:38.134110 5435 consensus_peers.cc:487] T 3e2b11c6788e4ccdbdde60769fee1aae P bf0be937c3634bcab62f1e58402a776f -> Peer f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133): Couldn't send request to peer f46fd51df5864af9b8493acd69db0747. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:35133: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250627 13:53:38.143304 4103 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:53:38.382882 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 3982
I20250627 13:53:38.425113 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 4112
I20250627 13:53:38.431017 4234 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:53:38.670367 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 4112
I20250627 13:53:38.713918 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 5159
I20250627 13:53:38.719551 5281 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:53:38.866371 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 5159
I20250627 13:53:38.893406 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 5290
I20250627 13:53:38.899446 5411 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:53:39.061831 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 5290
W20250627 13:53:39.090682 3778 connection.cc:537] server connection from 127.0.103.69:52581 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250627 13:53:39.090844 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 5420
I20250627 13:53:39.096552 5542 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:53:39.252326 6858 raft_consensus.cc:491] T 40b04be88e9947c5a5ca8432e3791bdd P bf0be937c3634bcab62f1e58402a776f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f46fd51df5864af9b8493acd69db0747)
I20250627 13:53:39.252561 6858 raft_consensus.cc:513] T 40b04be88e9947c5a5ca8432e3791bdd P bf0be937c3634bcab62f1e58402a776f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "f46fd51df5864af9b8493acd69db0747" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35133 } } peers { permanent_uuid: "bf0be937c3634bcab62f1e58402a776f" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 37215 } attrs { promote: false } }
I20250627 13:53:39.253427 6858 leader_election.cc:290] T 40b04be88e9947c5a5ca8432e3791bdd P bf0be937c3634bcab62f1e58402a776f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f46fd51df5864af9b8493acd69db0747 (127.0.103.65:35133)
I20250627 13:53:39.259398 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 5420
I20250627 13:53:39.284297 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 5550
I20250627 13:53:39.289729 5674 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:53:39.447980 7618 raft_consensus.cc:491] T 76518bc59dfe443abd8e97539d1bcb8a P 6503b59977444aaebe5ee70a0bc99154 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ad800aa8621b4fc5938692202990b5ab)
I20250627 13:53:39.448230 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 5550
I20250627 13:53:39.472527 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 6860
I20250627 13:53:39.478705 6983 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:53:39.598896 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 6860
I20250627 13:53:39.619565 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 3761
I20250627 13:53:39.620733 3822 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:53:39.783259 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 3761
2025-06-27T13:53:39Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/3 (50122 ms)
[----------] 1 test from RebalanceParamTest (50122 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1
I20250627 13:53:39.871703 413 test_util.cc:276] Using random seed: 23096265
I20250627 13:53:39.873445 413 ts_itest-base.cc:115] Starting cluster with:
I20250627 13:53:39.873498 413 ts_itest-base.cc:116] --------------
I20250627 13:53:39.873528 413 ts_itest-base.cc:117] 8 tablet servers
I20250627 13:53:39.873575 413 ts_itest-base.cc:118] 3 replicas per TS
I20250627 13:53:39.873613 413 ts_itest-base.cc:119] --------------
2025-06-27T13:53:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-27T13:53:39Z Disabled control of system clock
I20250627 13:53:39.916841 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.103.126:35849
--webserver_interface=127.0.103.126
--webserver_port=0
--builtin_ntp_servers=127.0.103.84:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.103.126:35849
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20250627 13:53:40.248728 7630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:53:40.249060 7630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:53:40.249198 7630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:53:40.256721 7630 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:53:40.256898 7630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250627 13:53:40.257011 7630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:53:40.257092 7630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250627 13:53:40.257187 7630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250627 13:53:40.266461 7630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.103.126:35849
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.103.126:35849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.103.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:53:40.268112 7630 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:53:40.269904 7630 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:53:40.280997 7636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:53:40.281028 7635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:53:40.281023 7638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:53:40.281407 7630 server_base.cc:1048] running on GCE node
I20250627 13:53:40.282323 7630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:53:40.283677 7630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:53:40.284883 7630 hybrid_clock.cc:648] HybridClock initialized: now 1751032420284823 us; error 38 us; skew 500 ppm
I20250627 13:53:40.285185 7630 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:53:40.287403 7630 webserver.cc:469] Webserver started at http://127.0.103.126:38955/ using document root <none> and password file <none>
I20250627 13:53:40.288071 7630 fs_manager.cc:362] Metadata directory not provided
I20250627 13:53:40.288187 7630 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:53:40.288465 7630 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:53:40.290766 7630 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e7a954e8f6cf484a984a188e218b202d"
format_stamp: "Formatted at 2025-06-27 13:53:40 on dist-test-slave-1ckg"
I20250627 13:53:40.291455 7630 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e7a954e8f6cf484a984a188e218b202d"
format_stamp: "Formatted at 2025-06-27 13:53:40 on dist-test-slave-1ckg"
I20250627 13:53:40.296569 7630 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20250627 13:53:40.299906 7644 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:40.301165 7630 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250627 13:53:40.301326 7630 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e7a954e8f6cf484a984a188e218b202d"
format_stamp: "Formatted at 2025-06-27 13:53:40 on dist-test-slave-1ckg"
I20250627 13:53:40.301499 7630 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:53:40.316939 7630 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:53:40.317759 7630 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:53:40.318006 7630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:53:40.337199 7630 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:35849
I20250627 13:53:40.337224 7695 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:35849 every 8 connection(s)
I20250627 13:53:40.338744 7630 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250627 13:53:40.342720 7696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:40.343573 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 7630
I20250627 13:53:40.343748 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250627 13:53:40.355576 7696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7a954e8f6cf484a984a188e218b202d: Bootstrap starting.
I20250627 13:53:40.359570 7696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7a954e8f6cf484a984a188e218b202d: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:40.360668 7696 log.cc:826] T 00000000000000000000000000000000 P e7a954e8f6cf484a984a188e218b202d: Log is configured to *not* fsync() on all Append() calls
I20250627 13:53:40.363492 7696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7a954e8f6cf484a984a188e218b202d: No bootstrap required, opened a new log
I20250627 13:53:40.368834 7696 raft_consensus.cc:357] T 00000000000000000000000000000000 P e7a954e8f6cf484a984a188e218b202d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a954e8f6cf484a984a188e218b202d" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35849 } }
I20250627 13:53:40.369128 7696 raft_consensus.cc:383] T 00000000000000000000000000000000 P e7a954e8f6cf484a984a188e218b202d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:40.369216 7696 raft_consensus.cc:738] T 00000000000000000000000000000000 P e7a954e8f6cf484a984a188e218b202d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a954e8f6cf484a984a188e218b202d, State: Initialized, Role: FOLLOWER
I20250627 13:53:40.369714 7696 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7a954e8f6cf484a984a188e218b202d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a954e8f6cf484a984a188e218b202d" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35849 } }
I20250627 13:53:40.369892 7696 raft_consensus.cc:397] T 00000000000000000000000000000000 P e7a954e8f6cf484a984a188e218b202d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:53:40.369968 7696 raft_consensus.cc:491] T 00000000000000000000000000000000 P e7a954e8f6cf484a984a188e218b202d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:53:40.370142 7696 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e7a954e8f6cf484a984a188e218b202d [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:40.372076 7696 raft_consensus.cc:513] T 00000000000000000000000000000000 P e7a954e8f6cf484a984a188e218b202d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a954e8f6cf484a984a188e218b202d" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35849 } }
I20250627 13:53:40.372524 7696 leader_election.cc:304] T 00000000000000000000000000000000 P e7a954e8f6cf484a984a188e218b202d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7a954e8f6cf484a984a188e218b202d; no voters:
I20250627 13:53:40.373153 7696 leader_election.cc:290] T 00000000000000000000000000000000 P e7a954e8f6cf484a984a188e218b202d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250627 13:53:40.373274 7701 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e7a954e8f6cf484a984a188e218b202d [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:40.373891 7701 raft_consensus.cc:695] T 00000000000000000000000000000000 P e7a954e8f6cf484a984a188e218b202d [term 1 LEADER]: Becoming Leader. State: Replica: e7a954e8f6cf484a984a188e218b202d, State: Running, Role: LEADER
I20250627 13:53:40.374437 7701 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7a954e8f6cf484a984a188e218b202d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a954e8f6cf484a984a188e218b202d" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35849 } }
I20250627 13:53:40.375365 7696 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7a954e8f6cf484a984a188e218b202d [sys.catalog]: configured and running, proceeding with master startup.
I20250627 13:53:40.377759 7702 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7a954e8f6cf484a984a188e218b202d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7a954e8f6cf484a984a188e218b202d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a954e8f6cf484a984a188e218b202d" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35849 } } }
I20250627 13:53:40.378055 7702 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7a954e8f6cf484a984a188e218b202d [sys.catalog]: This master's current role is: LEADER
I20250627 13:53:40.378111 7703 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7a954e8f6cf484a984a188e218b202d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7a954e8f6cf484a984a188e218b202d. Latest consensus state: current_term: 1 leader_uuid: "e7a954e8f6cf484a984a188e218b202d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a954e8f6cf484a984a188e218b202d" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 35849 } } }
I20250627 13:53:40.378443 7703 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7a954e8f6cf484a984a188e218b202d [sys.catalog]: This master's current role is: LEADER
I20250627 13:53:40.380201 7709 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250627 13:53:40.385669 7709 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250627 13:53:40.394308 7709 catalog_manager.cc:1349] Generated new cluster ID: a8e08adbe6994199bb46ab1a4135bd0c
I20250627 13:53:40.394439 7709 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250627 13:53:40.449232 7709 catalog_manager.cc:1372] Generated new certificate authority record
I20250627 13:53:40.450506 7709 catalog_manager.cc:1506] Loading token signing keys...
I20250627 13:53:40.461396 7709 catalog_manager.cc:5955] T 00000000000000000000000000000000 P e7a954e8f6cf484a984a188e218b202d: Generated new TSK 0
I20250627 13:53:40.462455 7709 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250627 13:53:40.477150 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.65:0
--local_ip_for_outbound_sockets=127.0.103.65
--webserver_interface=127.0.103.65
--webserver_port=0
--tserver_master_addrs=127.0.103.126:35849
--builtin_ntp_servers=127.0.103.84:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250627 13:53:40.795725 7720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:53:40.795998 7720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:53:40.796111 7720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:53:40.803488 7720 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:53:40.803584 7720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:53:40.803758 7720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.65
I20250627 13:53:40.812340 7720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.103.65
--webserver_port=0
--tserver_master_addrs=127.0.103.126:35849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.65
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:53:40.813882 7720 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:53:40.815681 7720 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:53:40.826925 7726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:53:40.826889 7728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:53:40.826856 7725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:53:40.827565 7720 server_base.cc:1048] running on GCE node
I20250627 13:53:40.828132 7720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:53:40.829039 7720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:53:40.830269 7720 hybrid_clock.cc:648] HybridClock initialized: now 1751032420830207 us; error 41 us; skew 500 ppm
I20250627 13:53:40.830535 7720 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:53:40.832792 7720 webserver.cc:469] Webserver started at http://127.0.103.65:42675/ using document root <none> and password file <none>
I20250627 13:53:40.833493 7720 fs_manager.cc:362] Metadata directory not provided
I20250627 13:53:40.833612 7720 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:53:40.833959 7720 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:53:40.836226 7720 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "83574c1bf8ea45adaff939158f93c096"
format_stamp: "Formatted at 2025-06-27 13:53:40 on dist-test-slave-1ckg"
I20250627 13:53:40.836943 7720 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "83574c1bf8ea45adaff939158f93c096"
format_stamp: "Formatted at 2025-06-27 13:53:40 on dist-test-slave-1ckg"
I20250627 13:53:40.841990 7720 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20250627 13:53:40.845162 7734 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:40.846383 7720 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250627 13:53:40.846529 7720 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "83574c1bf8ea45adaff939158f93c096"
format_stamp: "Formatted at 2025-06-27 13:53:40 on dist-test-slave-1ckg"
I20250627 13:53:40.846695 7720 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:53:40.866822 7720 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:53:40.867570 7720 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:53:40.867820 7720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:53:40.868688 7720 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:53:40.870141 7720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:53:40.870226 7720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:40.870340 7720 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:53:40.870393 7720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:40.901784 7720 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:37467
I20250627 13:53:40.901847 7846 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:37467 every 8 connection(s)
I20250627 13:53:40.903079 7720 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250627 13:53:40.903421 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 7720
I20250627 13:53:40.903600 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250627 13:53:40.907459 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.66:0
--local_ip_for_outbound_sockets=127.0.103.66
--webserver_interface=127.0.103.66
--webserver_port=0
--tserver_master_addrs=127.0.103.126:35849
--builtin_ntp_servers=127.0.103.84:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:53:40.917053 7847 heartbeater.cc:344] Connected to a master server at 127.0.103.126:35849
I20250627 13:53:40.917418 7847 heartbeater.cc:461] Registering TS with master...
I20250627 13:53:40.918320 7847 heartbeater.cc:507] Master 127.0.103.126:35849 requested a full tablet report, sending...
I20250627 13:53:40.920300 7661 ts_manager.cc:194] Registered new tserver with Master: 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:53:40.921811 7661 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.65:46151
W20250627 13:53:41.216572 7851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:53:41.216881 7851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:53:41.217015 7851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:53:41.225033 7851 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:53:41.225157 7851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:53:41.225345 7851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.66
I20250627 13:53:41.234498 7851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.103.66
--webserver_port=0
--tserver_master_addrs=127.0.103.126:35849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.66
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:53:41.236105 7851 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:53:41.237898 7851 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:53:41.248957 7859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:53:41.249017 7857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:53:41.248980 7856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:53:41.249761 7851 server_base.cc:1048] running on GCE node
I20250627 13:53:41.250311 7851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:53:41.251114 7851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:53:41.252323 7851 hybrid_clock.cc:648] HybridClock initialized: now 1751032421252247 us; error 50 us; skew 500 ppm
I20250627 13:53:41.252583 7851 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:53:41.254704 7851 webserver.cc:469] Webserver started at http://127.0.103.66:39393/ using document root <none> and password file <none>
I20250627 13:53:41.255359 7851 fs_manager.cc:362] Metadata directory not provided
I20250627 13:53:41.255474 7851 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:53:41.255774 7851 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:53:41.260076 7851 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
format_stamp: "Formatted at 2025-06-27 13:53:41 on dist-test-slave-1ckg"
I20250627 13:53:41.260932 7851 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
format_stamp: "Formatted at 2025-06-27 13:53:41 on dist-test-slave-1ckg"
I20250627 13:53:41.266062 7851 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20250627 13:53:41.269315 7865 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:41.270639 7851 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250627 13:53:41.270799 7851 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
format_stamp: "Formatted at 2025-06-27 13:53:41 on dist-test-slave-1ckg"
I20250627 13:53:41.270967 7851 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:53:41.287091 7851 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:53:41.287846 7851 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:53:41.288100 7851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:53:41.289036 7851 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:53:41.290779 7851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:53:41.290865 7851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:41.290988 7851 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:53:41.291044 7851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:41.323221 7851 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:43675
I20250627 13:53:41.323267 7977 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:43675 every 8 connection(s)
I20250627 13:53:41.324496 7851 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250627 13:53:41.334755 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 7851
I20250627 13:53:41.334924 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250627 13:53:41.336272 7978 heartbeater.cc:344] Connected to a master server at 127.0.103.126:35849
I20250627 13:53:41.336596 7978 heartbeater.cc:461] Registering TS with master...
I20250627 13:53:41.337524 7978 heartbeater.cc:507] Master 127.0.103.126:35849 requested a full tablet report, sending...
I20250627 13:53:41.338331 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.67:0
--local_ip_for_outbound_sockets=127.0.103.67
--webserver_interface=127.0.103.67
--webserver_port=0
--tserver_master_addrs=127.0.103.126:35849
--builtin_ntp_servers=127.0.103.84:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:53:41.339339 7661 ts_manager.cc:194] Registered new tserver with Master: a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:53:41.340293 7661 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.66:47419
W20250627 13:53:41.654734 7981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:53:41.655036 7981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:53:41.655179 7981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:53:41.662770 7981 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:53:41.662878 7981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:53:41.663094 7981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.67
I20250627 13:53:41.671867 7981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.103.67
--webserver_port=0
--tserver_master_addrs=127.0.103.126:35849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.67
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:53:41.673430 7981 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:53:41.675189 7981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:53:41.685807 7989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:53:41.685895 7986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:53:41.686029 7987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:53:41.686784 7981 server_base.cc:1048] running on GCE node
I20250627 13:53:41.687281 7981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:53:41.688086 7981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:53:41.689283 7981 hybrid_clock.cc:648] HybridClock initialized: now 1751032421689224 us; error 36 us; skew 500 ppm
I20250627 13:53:41.689532 7981 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:53:41.691557 7981 webserver.cc:469] Webserver started at http://127.0.103.67:46215/ using document root <none> and password file <none>
I20250627 13:53:41.692198 7981 fs_manager.cc:362] Metadata directory not provided
I20250627 13:53:41.692322 7981 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:53:41.692641 7981 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:53:41.695454 7981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9cf479bdcd0242738d2a77db4ad9f660"
format_stamp: "Formatted at 2025-06-27 13:53:41 on dist-test-slave-1ckg"
I20250627 13:53:41.696097 7981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9cf479bdcd0242738d2a77db4ad9f660"
format_stamp: "Formatted at 2025-06-27 13:53:41 on dist-test-slave-1ckg"
I20250627 13:53:41.700837 7981 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250627 13:53:41.703785 7995 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:41.704927 7981 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250627 13:53:41.705063 7981 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9cf479bdcd0242738d2a77db4ad9f660"
format_stamp: "Formatted at 2025-06-27 13:53:41 on dist-test-slave-1ckg"
I20250627 13:53:41.705221 7981 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:53:41.722072 7981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:53:41.722755 7981 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:53:41.722987 7981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:53:41.723814 7981 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:53:41.725239 7981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:53:41.725322 7981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:41.725440 7981 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:53:41.725494 7981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:41.756150 7981 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:38933
I20250627 13:53:41.756191 8107 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:38933 every 8 connection(s)
I20250627 13:53:41.757377 7981 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250627 13:53:41.758496 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 7981
I20250627 13:53:41.758697 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250627 13:53:41.762259 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.68:0
--local_ip_for_outbound_sockets=127.0.103.68
--webserver_interface=127.0.103.68
--webserver_port=0
--tserver_master_addrs=127.0.103.126:35849
--builtin_ntp_servers=127.0.103.84:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:53:41.771817 8108 heartbeater.cc:344] Connected to a master server at 127.0.103.126:35849
I20250627 13:53:41.772192 8108 heartbeater.cc:461] Registering TS with master...
I20250627 13:53:41.773062 8108 heartbeater.cc:507] Master 127.0.103.126:35849 requested a full tablet report, sending...
I20250627 13:53:41.774616 7661 ts_manager.cc:194] Registered new tserver with Master: 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:53:41.775720 7661 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.67:53405
I20250627 13:53:41.924708 7847 heartbeater.cc:499] Master 127.0.103.126:35849 was elected leader, sending a full tablet report...
W20250627 13:53:42.074931 8111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:53:42.075225 8111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:53:42.075347 8111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:53:42.082684 8111 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:53:42.082779 8111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:53:42.082943 8111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.68
I20250627 13:53:42.091651 8111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.103.68
--webserver_port=0
--tserver_master_addrs=127.0.103.126:35849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.68
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:53:42.093109 8111 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:53:42.094846 8111 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:53:42.105578 8119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:53:42.105609 8117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:53:42.105584 8116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:53:42.106585 8111 server_base.cc:1048] running on GCE node
I20250627 13:53:42.107123 8111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:53:42.107936 8111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:53:42.109148 8111 hybrid_clock.cc:648] HybridClock initialized: now 1751032422109065 us; error 40 us; skew 500 ppm
I20250627 13:53:42.109427 8111 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:53:42.111572 8111 webserver.cc:469] Webserver started at http://127.0.103.68:36495/ using document root <none> and password file <none>
I20250627 13:53:42.112233 8111 fs_manager.cc:362] Metadata directory not provided
I20250627 13:53:42.112372 8111 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:53:42.112694 8111 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:53:42.114941 8111 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e4c831df4da841929d13d6c783099b0e"
format_stamp: "Formatted at 2025-06-27 13:53:42 on dist-test-slave-1ckg"
I20250627 13:53:42.115607 8111 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e4c831df4da841929d13d6c783099b0e"
format_stamp: "Formatted at 2025-06-27 13:53:42 on dist-test-slave-1ckg"
I20250627 13:53:42.120520 8111 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20250627 13:53:42.123478 8125 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:42.124626 8111 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250627 13:53:42.124761 8111 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e4c831df4da841929d13d6c783099b0e"
format_stamp: "Formatted at 2025-06-27 13:53:42 on dist-test-slave-1ckg"
I20250627 13:53:42.124917 8111 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:53:42.136693 8111 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:53:42.137358 8111 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:53:42.137594 8111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:53:42.138391 8111 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:53:42.139777 8111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:53:42.139854 8111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:42.139973 8111 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:53:42.140028 8111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:42.176348 8111 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:46139
I20250627 13:53:42.176412 8237 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:46139 every 8 connection(s)
I20250627 13:53:42.177706 8111 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250627 13:53:42.180553 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 8111
I20250627 13:53:42.180701 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250627 13:53:42.183461 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.69:0
--local_ip_for_outbound_sockets=127.0.103.69
--webserver_interface=127.0.103.69
--webserver_port=0
--tserver_master_addrs=127.0.103.126:35849
--builtin_ntp_servers=127.0.103.84:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:53:42.190186 8238 heartbeater.cc:344] Connected to a master server at 127.0.103.126:35849
I20250627 13:53:42.190574 8238 heartbeater.cc:461] Registering TS with master...
I20250627 13:53:42.191560 8238 heartbeater.cc:507] Master 127.0.103.126:35849 requested a full tablet report, sending...
I20250627 13:53:42.193032 7661 ts_manager.cc:194] Registered new tserver with Master: e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:53:42.193828 7661 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.68:60223
I20250627 13:53:42.343120 7978 heartbeater.cc:499] Master 127.0.103.126:35849 was elected leader, sending a full tablet report...
W20250627 13:53:42.491992 8241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:53:42.492277 8241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:53:42.492381 8241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:53:42.499717 8241 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:53:42.499809 8241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:53:42.499927 8241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.69
I20250627 13:53:42.508646 8241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.103.69
--webserver_port=0
--tserver_master_addrs=127.0.103.126:35849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.69
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:53:42.510141 8241 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:53:42.511771 8241 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:53:42.522239 8247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:53:42.522413 8249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:53:42.522261 8246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:53:42.523118 8241 server_base.cc:1048] running on GCE node
I20250627 13:53:42.523639 8241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:53:42.524441 8241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:53:42.525636 8241 hybrid_clock.cc:648] HybridClock initialized: now 1751032422525573 us; error 44 us; skew 500 ppm
I20250627 13:53:42.525914 8241 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:53:42.528033 8241 webserver.cc:469] Webserver started at http://127.0.103.69:35087/ using document root <none> and password file <none>
I20250627 13:53:42.528643 8241 fs_manager.cc:362] Metadata directory not provided
I20250627 13:53:42.528749 8241 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:53:42.529036 8241 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:53:42.531297 8241 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "2a1282ebe92a4c9a92eed3b13f69012a"
format_stamp: "Formatted at 2025-06-27 13:53:42 on dist-test-slave-1ckg"
I20250627 13:53:42.532087 8241 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "2a1282ebe92a4c9a92eed3b13f69012a"
format_stamp: "Formatted at 2025-06-27 13:53:42 on dist-test-slave-1ckg"
I20250627 13:53:42.537034 8241 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250627 13:53:42.540401 8255 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:42.541743 8241 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250627 13:53:42.541891 8241 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "2a1282ebe92a4c9a92eed3b13f69012a"
format_stamp: "Formatted at 2025-06-27 13:53:42 on dist-test-slave-1ckg"
I20250627 13:53:42.542065 8241 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:53:42.569844 8241 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:53:42.570658 8241 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:53:42.570912 8241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:53:42.571882 8241 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:53:42.573465 8241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:53:42.573558 8241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:42.573710 8241 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:53:42.573776 8241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:42.605479 8241 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.69:33101
I20250627 13:53:42.605522 8367 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.69:33101 every 8 connection(s)
I20250627 13:53:42.606839 8241 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250627 13:53:42.609233 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 8241
I20250627 13:53:42.609416 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250627 13:53:42.612676 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.70:0
--local_ip_for_outbound_sockets=127.0.103.70
--webserver_interface=127.0.103.70
--webserver_port=0
--tserver_master_addrs=127.0.103.126:35849
--builtin_ntp_servers=127.0.103.84:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:53:42.619654 8368 heartbeater.cc:344] Connected to a master server at 127.0.103.126:35849
I20250627 13:53:42.619987 8368 heartbeater.cc:461] Registering TS with master...
I20250627 13:53:42.620786 8368 heartbeater.cc:507] Master 127.0.103.126:35849 requested a full tablet report, sending...
I20250627 13:53:42.622227 7661 ts_manager.cc:194] Registered new tserver with Master: 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101)
I20250627 13:53:42.623360 7661 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.69:37609
I20250627 13:53:42.778247 8108 heartbeater.cc:499] Master 127.0.103.126:35849 was elected leader, sending a full tablet report...
W20250627 13:53:42.930384 8371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:53:42.930701 8371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:53:42.930831 8371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:53:42.938442 8371 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:53:42.938542 8371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:53:42.938706 8371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.70
I20250627 13:53:42.947943 8371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.103.70
--webserver_port=0
--tserver_master_addrs=127.0.103.126:35849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.70
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:53:42.949534 8371 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:53:42.951310 8371 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:53:42.962148 8379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:53:42.962198 8376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:53:42.962173 8377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:53:42.963325 8371 server_base.cc:1048] running on GCE node
I20250627 13:53:42.963904 8371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:53:42.964811 8371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:53:42.966022 8371 hybrid_clock.cc:648] HybridClock initialized: now 1751032422965945 us; error 44 us; skew 500 ppm
I20250627 13:53:42.966295 8371 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:53:42.968483 8371 webserver.cc:469] Webserver started at http://127.0.103.70:41307/ using document root <none> and password file <none>
I20250627 13:53:42.969152 8371 fs_manager.cc:362] Metadata directory not provided
I20250627 13:53:42.969266 8371 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:53:42.969606 8371 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:53:42.971889 8371 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "956ff058e6d64579a1963aabdb6d466b"
format_stamp: "Formatted at 2025-06-27 13:53:42 on dist-test-slave-1ckg"
I20250627 13:53:42.972651 8371 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "956ff058e6d64579a1963aabdb6d466b"
format_stamp: "Formatted at 2025-06-27 13:53:42 on dist-test-slave-1ckg"
I20250627 13:53:42.977792 8371 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20250627 13:53:42.981020 8385 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:42.982358 8371 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250627 13:53:42.982517 8371 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "956ff058e6d64579a1963aabdb6d466b"
format_stamp: "Formatted at 2025-06-27 13:53:42 on dist-test-slave-1ckg"
I20250627 13:53:42.982694 8371 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:53:43.017251 8371 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:53:43.018142 8371 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:53:43.018404 8371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:53:43.019359 8371 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:53:43.020854 8371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:53:43.020934 8371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:43.021009 8371 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:53:43.021049 8371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:43.052371 8371 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.70:40647
I20250627 13:53:43.052440 8497 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.70:40647 every 8 connection(s)
I20250627 13:53:43.053824 8371 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20250627 13:53:43.059329 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 8371
I20250627 13:53:43.059502 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20250627 13:53:43.062516 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.71:0
--local_ip_for_outbound_sockets=127.0.103.71
--webserver_interface=127.0.103.71
--webserver_port=0
--tserver_master_addrs=127.0.103.126:35849
--builtin_ntp_servers=127.0.103.84:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:53:43.066201 8498 heartbeater.cc:344] Connected to a master server at 127.0.103.126:35849
I20250627 13:53:43.066550 8498 heartbeater.cc:461] Registering TS with master...
I20250627 13:53:43.067313 8498 heartbeater.cc:507] Master 127.0.103.126:35849 requested a full tablet report, sending...
I20250627 13:53:43.068735 7661 ts_manager.cc:194] Registered new tserver with Master: 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647)
I20250627 13:53:43.069491 7661 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.70:40499
I20250627 13:53:43.196112 8238 heartbeater.cc:499] Master 127.0.103.126:35849 was elected leader, sending a full tablet report...
W20250627 13:53:43.381145 8501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:53:43.381453 8501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:53:43.381578 8501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:53:43.389950 8501 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:53:43.390080 8501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:53:43.390264 8501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.71
I20250627 13:53:43.398928 8501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.103.71
--webserver_port=0
--tserver_master_addrs=127.0.103.126:35849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.71
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:53:43.400485 8501 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:53:43.402292 8501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:53:43.413172 8509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:53:43.413115 8506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:53:43.413142 8507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:53:43.413846 8501 server_base.cc:1048] running on GCE node
I20250627 13:53:43.414330 8501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:53:43.415169 8501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:53:43.416376 8501 hybrid_clock.cc:648] HybridClock initialized: now 1751032423416316 us; error 32 us; skew 500 ppm
I20250627 13:53:43.416657 8501 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:53:43.421761 8501 webserver.cc:469] Webserver started at http://127.0.103.71:36169/ using document root <none> and password file <none>
I20250627 13:53:43.422426 8501 fs_manager.cc:362] Metadata directory not provided
I20250627 13:53:43.422535 8501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:53:43.422849 8501 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:53:43.425180 8501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "6686c6c365bf4f4582058890161676fe"
format_stamp: "Formatted at 2025-06-27 13:53:43 on dist-test-slave-1ckg"
I20250627 13:53:43.425930 8501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "6686c6c365bf4f4582058890161676fe"
format_stamp: "Formatted at 2025-06-27 13:53:43 on dist-test-slave-1ckg"
I20250627 13:53:43.430896 8501 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20250627 13:53:43.433815 8515 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:43.434996 8501 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250627 13:53:43.435134 8501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "6686c6c365bf4f4582058890161676fe"
format_stamp: "Formatted at 2025-06-27 13:53:43 on dist-test-slave-1ckg"
I20250627 13:53:43.435297 8501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:53:43.452162 8501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:53:43.452956 8501 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:53:43.453202 8501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:53:43.454169 8501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:53:43.455724 8501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:53:43.455790 8501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:43.455895 8501 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:53:43.455940 8501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:43.489408 8501 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.71:37859
I20250627 13:53:43.489509 8627 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.71:37859 every 8 connection(s)
I20250627 13:53:43.490792 8501 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20250627 13:53:43.499850 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 8501
I20250627 13:53:43.500011 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20250627 13:53:43.503319 8628 heartbeater.cc:344] Connected to a master server at 127.0.103.126:35849
I20250627 13:53:43.503405 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.72:0
--local_ip_for_outbound_sockets=127.0.103.72
--webserver_interface=127.0.103.72
--webserver_port=0
--tserver_master_addrs=127.0.103.126:35849
--builtin_ntp_servers=127.0.103.84:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:53:43.503684 8628 heartbeater.cc:461] Registering TS with master...
I20250627 13:53:43.504484 8628 heartbeater.cc:507] Master 127.0.103.126:35849 requested a full tablet report, sending...
I20250627 13:53:43.505937 7661 ts_manager.cc:194] Registered new tserver with Master: 6686c6c365bf4f4582058890161676fe (127.0.103.71:37859)
I20250627 13:53:43.507094 7661 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.71:52955
I20250627 13:53:43.625797 8368 heartbeater.cc:499] Master 127.0.103.126:35849 was elected leader, sending a full tablet report...
W20250627 13:53:43.822197 8631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:53:43.822508 8631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:53:43.822633 8631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:53:43.830087 8631 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:53:43.830178 8631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:53:43.830324 8631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.72
I20250627 13:53:43.839125 8631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.103.72
--webserver_port=0
--tserver_master_addrs=127.0.103.126:35849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.72
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:53:43.840595 8631 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:53:43.842358 8631 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:53:43.853826 8636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:53:43.853853 8639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:53:43.853811 8637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:53:43.854856 8631 server_base.cc:1048] running on GCE node
I20250627 13:53:43.855350 8631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:53:43.856117 8631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:53:43.857347 8631 hybrid_clock.cc:648] HybridClock initialized: now 1751032423857247 us; error 72 us; skew 500 ppm
I20250627 13:53:43.857616 8631 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:53:43.859755 8631 webserver.cc:469] Webserver started at http://127.0.103.72:43603/ using document root <none> and password file <none>
I20250627 13:53:43.860427 8631 fs_manager.cc:362] Metadata directory not provided
I20250627 13:53:43.860544 8631 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:53:43.860868 8631 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:53:43.863067 8631 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "dede81a8c0164427a61532d01b6fc157"
format_stamp: "Formatted at 2025-06-27 13:53:43 on dist-test-slave-1ckg"
I20250627 13:53:43.863791 8631 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "dede81a8c0164427a61532d01b6fc157"
format_stamp: "Formatted at 2025-06-27 13:53:43 on dist-test-slave-1ckg"
I20250627 13:53:43.868803 8631 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20250627 13:53:43.871788 8645 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:43.872954 8631 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250627 13:53:43.873095 8631 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "dede81a8c0164427a61532d01b6fc157"
format_stamp: "Formatted at 2025-06-27 13:53:43 on dist-test-slave-1ckg"
I20250627 13:53:43.873304 8631 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:53:43.887857 8631 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:53:43.888566 8631 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:53:43.888806 8631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:53:43.889628 8631 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:53:43.891072 8631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:53:43.891140 8631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:43.891254 8631 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:53:43.891302 8631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:53:43.920181 8631 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.72:35213
I20250627 13:53:43.920215 8757 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.72:35213 every 8 connection(s)
I20250627 13:53:43.921566 8631 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20250627 13:53:43.929469 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 8631
I20250627 13:53:43.929719 413 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20250627 13:53:43.933946 8758 heartbeater.cc:344] Connected to a master server at 127.0.103.126:35849
I20250627 13:53:43.934230 8758 heartbeater.cc:461] Registering TS with master...
I20250627 13:53:43.934984 8758 heartbeater.cc:507] Master 127.0.103.126:35849 requested a full tablet report, sending...
I20250627 13:53:43.936141 7661 ts_manager.cc:194] Registered new tserver with Master: dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213)
I20250627 13:53:43.936789 7661 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.72:54477
I20250627 13:53:43.944929 413 external_mini_cluster.cc:934] 8 TS(s) registered with all masters
I20250627 13:53:43.960602 413 external_mini_cluster.cc:934] 8 TS(s) registered with all masters
I20250627 13:53:43.960778 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 8241
I20250627 13:53:43.968645 8363 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:53:44.044966 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 8241
I20250627 13:53:44.060899 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 8371
I20250627 13:53:44.068468 8493 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:53:44.144922 8498 heartbeater.cc:499] Master 127.0.103.126:35849 was elected leader, sending a full tablet report...
I20250627 13:53:44.146535 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 8371
I20250627 13:53:44.163918 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 8501
I20250627 13:53:44.172194 8623 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:53:44.251029 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 8501
I20250627 13:53:44.266867 413 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu pid 8631
I20250627 13:53:44.274955 8753 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:53:44.352173 413 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu with pid 8631
I20250627 13:53:48.119498 7661 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45288:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250627 13:53:48.175653 8173 tablet_service.cc:1468] Processing CreateTablet for tablet e3053b529a264e8697a1df9e17eeea81 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d8bfc9268a4a4c2c92bc20d56ac73086]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:48.177325 8173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3053b529a264e8697a1df9e17eeea81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:48.177261 7912 tablet_service.cc:1468] Processing CreateTablet for tablet b265cfda4bfe48908659208ec8eeab2d (DEFAULT_TABLE table=rebalance_test_table_0 [id=d8bfc9268a4a4c2c92bc20d56ac73086]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:48.178615 7912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b265cfda4bfe48908659208ec8eeab2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:48.179811 7782 tablet_service.cc:1468] Processing CreateTablet for tablet e3053b529a264e8697a1df9e17eeea81 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d8bfc9268a4a4c2c92bc20d56ac73086]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:48.181038 7782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3053b529a264e8697a1df9e17eeea81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:48.181996 8172 tablet_service.cc:1468] Processing CreateTablet for tablet b265cfda4bfe48908659208ec8eeab2d (DEFAULT_TABLE table=rebalance_test_table_0 [id=d8bfc9268a4a4c2c92bc20d56ac73086]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:48.182475 8172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b265cfda4bfe48908659208ec8eeab2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:48.189579 8043 tablet_service.cc:1468] Processing CreateTablet for tablet e3053b529a264e8697a1df9e17eeea81 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d8bfc9268a4a4c2c92bc20d56ac73086]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:48.190340 7781 tablet_service.cc:1468] Processing CreateTablet for tablet d2eb537d49804e1b8ec5a217b1564988 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d8bfc9268a4a4c2c92bc20d56ac73086]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:48.190752 7781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2eb537d49804e1b8ec5a217b1564988. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:48.191406 8043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3053b529a264e8697a1df9e17eeea81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:48.190562 8042 tablet_service.cc:1468] Processing CreateTablet for tablet d2eb537d49804e1b8ec5a217b1564988 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d8bfc9268a4a4c2c92bc20d56ac73086]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:48.193995 8042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2eb537d49804e1b8ec5a217b1564988. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:48.194641 7913 tablet_service.cc:1468] Processing CreateTablet for tablet d2eb537d49804e1b8ec5a217b1564988 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d8bfc9268a4a4c2c92bc20d56ac73086]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:48.195137 7913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2eb537d49804e1b8ec5a217b1564988. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:48.198199 7780 tablet_service.cc:1468] Processing CreateTablet for tablet b265cfda4bfe48908659208ec8eeab2d (DEFAULT_TABLE table=rebalance_test_table_0 [id=d8bfc9268a4a4c2c92bc20d56ac73086]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:48.198602 7780 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b265cfda4bfe48908659208ec8eeab2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:48.233541 8790 tablet_bootstrap.cc:492] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap starting.
I20250627 13:53:48.239113 8789 tablet_bootstrap.cc:492] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:53:48.239698 8790 tablet_bootstrap.cc:654] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:48.239892 8787 tablet_bootstrap.cc:492] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
I20250627 13:53:48.239892 8788 tablet_bootstrap.cc:492] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:53:48.241129 8790 log.cc:826] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660: Log is configured to *not* fsync() on all Append() calls
I20250627 13:53:48.242810 8789 tablet_bootstrap.cc:654] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:48.243276 8787 tablet_bootstrap.cc:654] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:48.243487 8788 tablet_bootstrap.cc:654] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:48.244026 8789 log.cc:826] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: Log is configured to *not* fsync() on all Append() calls
I20250627 13:53:48.244464 8787 log.cc:826] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: Log is configured to *not* fsync() on all Append() calls
I20250627 13:53:48.244940 8788 log.cc:826] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854: Log is configured to *not* fsync() on all Append() calls
I20250627 13:53:48.252178 8789 tablet_bootstrap.cc:492] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: No bootstrap required, opened a new log
I20250627 13:53:48.252192 8790 tablet_bootstrap.cc:492] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660: No bootstrap required, opened a new log
I20250627 13:53:48.252558 8789 ts_tablet_manager.cc:1397] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20250627 13:53:48.252561 8790 ts_tablet_manager.cc:1397] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.004s
I20250627 13:53:48.263059 8790 raft_consensus.cc:357] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:48.263587 8790 raft_consensus.cc:383] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:48.263751 8790 raft_consensus.cc:738] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Initialized, Role: FOLLOWER
I20250627 13:53:48.263844 8789 raft_consensus.cc:357] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:48.264214 8789 raft_consensus.cc:383] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:48.264400 8789 raft_consensus.cc:738] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: FOLLOWER
I20250627 13:53:48.264741 8790 consensus_queue.cc:260] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:48.265067 8789 consensus_queue.cc:260] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:48.265373 8788 tablet_bootstrap.cc:492] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854: No bootstrap required, opened a new log
I20250627 13:53:48.265372 8787 tablet_bootstrap.cc:492] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: No bootstrap required, opened a new log
I20250627 13:53:48.265718 8787 ts_tablet_manager.cc:1397] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.012s
I20250627 13:53:48.265707 8788 ts_tablet_manager.cc:1397] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.017s
I20250627 13:53:48.272404 8788 raft_consensus.cc:357] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:48.272791 8788 raft_consensus.cc:383] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:48.272982 8788 raft_consensus.cc:738] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: FOLLOWER
I20250627 13:53:48.273320 8787 raft_consensus.cc:357] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:48.273723 8787 raft_consensus.cc:383] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:48.273892 8787 raft_consensus.cc:738] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: FOLLOWER
I20250627 13:53:48.273743 8788 consensus_queue.cc:260] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:48.274091 8790 ts_tablet_manager.cc:1428] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent starting tablet: real 0.021s user 0.018s sys 0.000s
I20250627 13:53:48.274662 8787 consensus_queue.cc:260] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:48.275950 8789 ts_tablet_manager.cc:1428] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.023s user 0.015s sys 0.005s
I20250627 13:53:48.276618 8790 tablet_bootstrap.cc:492] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap starting.
I20250627 13:53:48.278512 8790 tablet_bootstrap.cc:654] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:48.280045 8789 tablet_bootstrap.cc:492] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:53:48.280084 8787 ts_tablet_manager.cc:1428] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.014s user 0.006s sys 0.005s
I20250627 13:53:48.280645 8787 tablet_bootstrap.cc:492] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
I20250627 13:53:48.282819 8789 tablet_bootstrap.cc:654] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:48.283093 8787 tablet_bootstrap.cc:654] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:48.286000 8788 ts_tablet_manager.cc:1428] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.020s user 0.008s sys 0.010s
I20250627 13:53:48.287155 8790 tablet_bootstrap.cc:492] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660: No bootstrap required, opened a new log
I20250627 13:53:48.287358 8790 ts_tablet_manager.cc:1397] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20250627 13:53:48.287503 8788 tablet_bootstrap.cc:492] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:53:48.288326 8790 raft_consensus.cc:357] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:48.288542 8790 raft_consensus.cc:383] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:48.288619 8790 raft_consensus.cc:738] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Initialized, Role: FOLLOWER
I20250627 13:53:48.288862 8790 consensus_queue.cc:260] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:48.289433 8790 ts_tablet_manager.cc:1428] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:53:48.290350 8787 tablet_bootstrap.cc:492] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e: No bootstrap required, opened a new log
I20250627 13:53:48.290529 8787 ts_tablet_manager.cc:1397] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20250627 13:53:48.291478 8787 raft_consensus.cc:357] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:48.291663 8787 raft_consensus.cc:383] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:48.291734 8787 raft_consensus.cc:738] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: FOLLOWER
I20250627 13:53:48.291990 8787 consensus_queue.cc:260] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:48.292243 8788 tablet_bootstrap.cc:654] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:48.292519 8787 ts_tablet_manager.cc:1428] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250627 13:53:48.293946 8789 tablet_bootstrap.cc:492] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096: No bootstrap required, opened a new log
I20250627 13:53:48.294121 8789 ts_tablet_manager.cc:1397] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20250627 13:53:48.294324 8788 tablet_bootstrap.cc:492] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854: No bootstrap required, opened a new log
I20250627 13:53:48.294531 8788 ts_tablet_manager.cc:1397] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20250627 13:53:48.294883 8789 raft_consensus.cc:357] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:48.295050 8789 raft_consensus.cc:383] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:48.295125 8789 raft_consensus.cc:738] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: FOLLOWER
I20250627 13:53:48.295351 8789 consensus_queue.cc:260] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:48.295482 8788 raft_consensus.cc:357] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:48.295698 8788 raft_consensus.cc:383] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:48.295747 8788 raft_consensus.cc:738] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: FOLLOWER
I20250627 13:53:48.295744 8789 ts_tablet_manager.cc:1428] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250627 13:53:48.295943 8788 consensus_queue.cc:260] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:48.296017 8789 tablet_bootstrap.cc:492] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:53:48.296869 8795 raft_consensus.cc:491] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:48.297098 8795 raft_consensus.cc:513] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:48.297931 8788 ts_tablet_manager.cc:1428] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250627 13:53:48.298909 8795 leader_election.cc:290] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675), 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:53:48.299059 8789 tablet_bootstrap.cc:654] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:48.301470 8789 tablet_bootstrap.cc:492] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096: No bootstrap required, opened a new log
I20250627 13:53:48.301585 8789 ts_tablet_manager.cc:1397] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250627 13:53:48.302114 8789 raft_consensus.cc:357] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:48.302268 8789 raft_consensus.cc:383] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:48.302366 8789 raft_consensus.cc:738] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: FOLLOWER
I20250627 13:53:48.302585 8789 consensus_queue.cc:260] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:48.303074 8789 ts_tablet_manager.cc:1428] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250627 13:53:48.309194 7933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2eb537d49804e1b8ec5a217b1564988" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" is_pre_election: true
I20250627 13:53:48.309440 7802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2eb537d49804e1b8ec5a217b1564988" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096" is_pre_election: true
I20250627 13:53:48.309743 7933 raft_consensus.cc:2466] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 0.
I20250627 13:53:48.309855 7802 raft_consensus.cc:2466] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 0.
I20250627 13:53:48.310379 7998 leader_election.cc:304] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, 9cf479bdcd0242738d2a77db4ad9f660, a41a89ae8ed743edb2b5cf4490d8c854; no voters:
I20250627 13:53:48.310752 8795 raft_consensus.cc:2802] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:48.310889 8795 raft_consensus.cc:491] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:48.310987 8795 raft_consensus.cc:3058] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:48.312827 8795 raft_consensus.cc:513] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:48.313341 8795 leader_election.cc:290] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 election: Requested vote from peers a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675), 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:53:48.313741 7933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2eb537d49804e1b8ec5a217b1564988" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
I20250627 13:53:48.313774 7802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d2eb537d49804e1b8ec5a217b1564988" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096"
I20250627 13:53:48.313890 7933 raft_consensus.cc:3058] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:48.313915 7802 raft_consensus.cc:3058] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:48.315990 7933 raft_consensus.cc:2466] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 1.
I20250627 13:53:48.316023 7802 raft_consensus.cc:2466] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 1.
I20250627 13:53:48.316356 7998 leader_election.cc:304] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cf479bdcd0242738d2a77db4ad9f660, a41a89ae8ed743edb2b5cf4490d8c854; no voters:
I20250627 13:53:48.316586 8795 raft_consensus.cc:2802] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:48.316939 8795 raft_consensus.cc:695] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Becoming Leader. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Running, Role: LEADER
I20250627 13:53:48.317341 8795 consensus_queue.cc:237] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:48.322108 7660 catalog_manager.cc:5582] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } }
W20250627 13:53:48.329929 7979 tablet.cc:2378] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:53:48.361699 8797 raft_consensus.cc:491] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:48.361891 8797 raft_consensus.cc:513] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:48.363482 8797 leader_election.cc:290] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:53:48.370198 8193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b265cfda4bfe48908659208ec8eeab2d" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e" is_pre_election: true
I20250627 13:53:48.370335 7802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b265cfda4bfe48908659208ec8eeab2d" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096" is_pre_election: true
I20250627 13:53:48.370551 7802 raft_consensus.cc:2466] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a41a89ae8ed743edb2b5cf4490d8c854 in term 0.
I20250627 13:53:48.370720 8193 raft_consensus.cc:2466] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a41a89ae8ed743edb2b5cf4490d8c854 in term 0.
I20250627 13:53:48.370958 7866 leader_election.cc:304] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, a41a89ae8ed743edb2b5cf4490d8c854; no voters:
I20250627 13:53:48.371349 8797 raft_consensus.cc:2802] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:48.371450 8797 raft_consensus.cc:491] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:48.371539 8797 raft_consensus.cc:3058] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:48.373442 8797 raft_consensus.cc:513] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:48.374051 8797 leader_election.cc:290] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 1 election: Requested vote from peers e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:53:48.374368 8193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b265cfda4bfe48908659208ec8eeab2d" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e"
I20250627 13:53:48.374527 7802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b265cfda4bfe48908659208ec8eeab2d" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096"
I20250627 13:53:48.374565 8193 raft_consensus.cc:3058] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:48.374727 7802 raft_consensus.cc:3058] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:48.377595 7802 raft_consensus.cc:2466] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41a89ae8ed743edb2b5cf4490d8c854 in term 1.
I20250627 13:53:48.377609 8193 raft_consensus.cc:2466] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41a89ae8ed743edb2b5cf4490d8c854 in term 1.
I20250627 13:53:48.378000 7869 leader_election.cc:304] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a41a89ae8ed743edb2b5cf4490d8c854, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:48.378270 8797 raft_consensus.cc:2802] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:48.378645 8797 raft_consensus.cc:695] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Becoming Leader. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Running, Role: LEADER
I20250627 13:53:48.379120 8797 consensus_queue.cc:237] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:48.383618 7660 catalog_manager.cc:5582] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: term changed from 0 to 1, leader changed from <none> to a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: HEALTHY } } }
W20250627 13:53:48.410429 7848 tablet.cc:2378] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250627 13:53:48.436112 8239 tablet.cc:2378] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:53:48.438014 8798 raft_consensus.cc:491] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:48.438177 8798 raft_consensus.cc:513] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:48.439762 8798 leader_election.cc:290] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467), 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:53:48.446892 7802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3053b529a264e8697a1df9e17eeea81" candidate_uuid: "e4c831df4da841929d13d6c783099b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096" is_pre_election: true
I20250627 13:53:48.447172 7802 raft_consensus.cc:2466] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4c831df4da841929d13d6c783099b0e in term 0.
I20250627 13:53:48.447594 8126 leader_election.cc:304] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:48.447547 8063 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3053b529a264e8697a1df9e17eeea81" candidate_uuid: "e4c831df4da841929d13d6c783099b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660" is_pre_election: true
I20250627 13:53:48.447902 8063 raft_consensus.cc:2466] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4c831df4da841929d13d6c783099b0e in term 0.
I20250627 13:53:48.447935 8798 raft_consensus.cc:2802] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:48.448060 8798 raft_consensus.cc:491] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:48.448163 8798 raft_consensus.cc:3058] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:48.450337 8798 raft_consensus.cc:513] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:48.450882 8798 leader_election.cc:290] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [CANDIDATE]: Term 1 election: Requested vote from peers 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467), 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:53:48.451335 7802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3053b529a264e8697a1df9e17eeea81" candidate_uuid: "e4c831df4da841929d13d6c783099b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096"
I20250627 13:53:48.451500 7802 raft_consensus.cc:3058] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:48.451452 8063 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3053b529a264e8697a1df9e17eeea81" candidate_uuid: "e4c831df4da841929d13d6c783099b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660"
I20250627 13:53:48.451628 8063 raft_consensus.cc:3058] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:48.453367 7802 raft_consensus.cc:2466] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c831df4da841929d13d6c783099b0e in term 1.
I20250627 13:53:48.453672 8063 raft_consensus.cc:2466] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c831df4da841929d13d6c783099b0e in term 1.
I20250627 13:53:48.453800 8126 leader_election.cc:304] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:48.454087 8798 raft_consensus.cc:2802] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:48.454490 8798 raft_consensus.cc:695] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Becoming Leader. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Running, Role: LEADER
I20250627 13:53:48.454980 8798 consensus_queue.cc:237] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:48.459326 7660 catalog_manager.cc:5582] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e reported cstate change: term changed from 0 to 1, leader changed from <none> to e4c831df4da841929d13d6c783099b0e (127.0.103.68). New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: UNKNOWN } } }
W20250627 13:53:48.513226 8109 tablet.cc:2378] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:53:48.703294 8795 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:48.709854 8795 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250627 13:53:48.818837 8816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:53:48.819157 8816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:53:48.826607 8816 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250627 13:53:48.840975 8797 consensus_queue.cc:1035] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:48.847792 8797 consensus_queue.cc:1035] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20250627 13:53:48.967732 8063 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250627 13:53:48.967815 7802 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250627 13:53:48.968735 8798 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:48.969311 8798 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:49.034802 8818 mvcc.cc:204] Tried to move back new op lower bound from 7172228829032235008 to 7172228826955489280. Current Snapshot: MvccSnapshot[applied={T|T < 7172228829032235008}]
I20250627 13:53:49.067395 8821 mvcc.cc:204] Tried to move back new op lower bound from 7172228829032235008 to 7172228826955489280. Current Snapshot: MvccSnapshot[applied={T|T < 7172228829032235008 or (T in {7172228829032235008})}]
INSERT report
rows total: 2000
time total: 236.174 ms
time per row: 0.118087 ms
I20250627 13:53:49.220119 7657 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45288:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250627 13:53:49.228755 8043 tablet_service.cc:1468] Processing CreateTablet for tablet 644985e3086d4614b6598a727cb60c1d (DEFAULT_TABLE table=rebalance_test_table_1 [id=8961aa0df2f1488f82cd55cfa22ad214]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:49.229203 8043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 644985e3086d4614b6598a727cb60c1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:49.229072 7912 tablet_service.cc:1468] Processing CreateTablet for tablet 644985e3086d4614b6598a727cb60c1d (DEFAULT_TABLE table=rebalance_test_table_1 [id=8961aa0df2f1488f82cd55cfa22ad214]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:49.229513 7912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 644985e3086d4614b6598a727cb60c1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:49.230050 8173 tablet_service.cc:1468] Processing CreateTablet for tablet 644985e3086d4614b6598a727cb60c1d (DEFAULT_TABLE table=rebalance_test_table_1 [id=8961aa0df2f1488f82cd55cfa22ad214]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:49.230446 8173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 644985e3086d4614b6598a727cb60c1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:49.234315 7913 tablet_service.cc:1468] Processing CreateTablet for tablet 03640a5124d2405faf4a26d0e22e1ffc (DEFAULT_TABLE table=rebalance_test_table_1 [id=8961aa0df2f1488f82cd55cfa22ad214]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:49.234390 7911 tablet_service.cc:1468] Processing CreateTablet for tablet 87726824b05a4b968386f57d3a7a4aad (DEFAULT_TABLE table=rebalance_test_table_1 [id=8961aa0df2f1488f82cd55cfa22ad214]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:49.234622 7781 tablet_service.cc:1468] Processing CreateTablet for tablet 87726824b05a4b968386f57d3a7a4aad (DEFAULT_TABLE table=rebalance_test_table_1 [id=8961aa0df2f1488f82cd55cfa22ad214]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:49.234764 7913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03640a5124d2405faf4a26d0e22e1ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:49.235028 7781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87726824b05a4b968386f57d3a7a4aad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:49.235095 7911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87726824b05a4b968386f57d3a7a4aad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:49.235499 8041 tablet_service.cc:1468] Processing CreateTablet for tablet 87726824b05a4b968386f57d3a7a4aad (DEFAULT_TABLE table=rebalance_test_table_1 [id=8961aa0df2f1488f82cd55cfa22ad214]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:49.235925 8041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87726824b05a4b968386f57d3a7a4aad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:49.238211 8172 tablet_service.cc:1468] Processing CreateTablet for tablet 03640a5124d2405faf4a26d0e22e1ffc (DEFAULT_TABLE table=rebalance_test_table_1 [id=8961aa0df2f1488f82cd55cfa22ad214]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:49.238595 8172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03640a5124d2405faf4a26d0e22e1ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:49.245596 8879 tablet_bootstrap.cc:492] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:53:49.245781 8042 tablet_service.cc:1468] Processing CreateTablet for tablet 03640a5124d2405faf4a26d0e22e1ffc (DEFAULT_TABLE table=rebalance_test_table_1 [id=8961aa0df2f1488f82cd55cfa22ad214]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:49.246165 8042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03640a5124d2405faf4a26d0e22e1ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:49.248710 8877 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap starting.
I20250627 13:53:49.250922 8880 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
I20250627 13:53:49.252794 8879 tablet_bootstrap.cc:654] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:49.253141 8880 tablet_bootstrap.cc:654] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:49.253698 8878 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:53:49.254730 8877 tablet_bootstrap.cc:654] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:49.256000 8878 tablet_bootstrap.cc:654] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:49.256546 8879 tablet_bootstrap.cc:492] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096: No bootstrap required, opened a new log
I20250627 13:53:49.256744 8879 ts_tablet_manager.cc:1397] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.001s
I20250627 13:53:49.257803 8879 raft_consensus.cc:357] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:49.258184 8879 raft_consensus.cc:383] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:49.258342 8879 raft_consensus.cc:738] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: FOLLOWER
I20250627 13:53:49.258682 8879 consensus_queue.cc:260] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:49.259454 8879 ts_tablet_manager.cc:1428] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250627 13:53:49.259722 8877 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660: No bootstrap required, opened a new log
I20250627 13:53:49.259912 8877 ts_tablet_manager.cc:1397] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.005s
I20250627 13:53:49.260790 8877 raft_consensus.cc:357] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:49.261139 8877 raft_consensus.cc:383] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:49.261305 8877 raft_consensus.cc:738] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Initialized, Role: FOLLOWER
I20250627 13:53:49.261458 8878 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854: No bootstrap required, opened a new log
I20250627 13:53:49.261670 8878 ts_tablet_manager.cc:1397] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20250627 13:53:49.261641 8877 consensus_queue.cc:260] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:49.262472 8878 raft_consensus.cc:357] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:49.262573 8877 ts_tablet_manager.cc:1428] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:53:49.262729 8878 raft_consensus.cc:383] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:49.262845 8877 tablet_bootstrap.cc:492] T 03640a5124d2405faf4a26d0e22e1ffc P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap starting.
I20250627 13:53:49.262874 8878 raft_consensus.cc:738] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: FOLLOWER
I20250627 13:53:49.263180 8878 consensus_queue.cc:260] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:49.264350 8880 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: No bootstrap required, opened a new log
I20250627 13:53:49.264529 8880 ts_tablet_manager.cc:1397] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20250627 13:53:49.264551 8877 tablet_bootstrap.cc:654] T 03640a5124d2405faf4a26d0e22e1ffc P 9cf479bdcd0242738d2a77db4ad9f660: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:49.264716 8878 ts_tablet_manager.cc:1428] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250627 13:53:49.265025 8878 tablet_bootstrap.cc:492] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:53:49.265388 8880 raft_consensus.cc:357] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:49.265595 8880 raft_consensus.cc:383] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:49.265749 8880 raft_consensus.cc:738] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: FOLLOWER
I20250627 13:53:49.266026 8880 consensus_queue.cc:260] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:49.267194 8880 ts_tablet_manager.cc:1428] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250627 13:53:49.267474 8880 tablet_bootstrap.cc:492] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
I20250627 13:53:49.267998 8877 tablet_bootstrap.cc:492] T 03640a5124d2405faf4a26d0e22e1ffc P 9cf479bdcd0242738d2a77db4ad9f660: No bootstrap required, opened a new log
I20250627 13:53:49.268944 8877 ts_tablet_manager.cc:1397] T 03640a5124d2405faf4a26d0e22e1ffc P 9cf479bdcd0242738d2a77db4ad9f660: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20250627 13:53:49.269309 8880 tablet_bootstrap.cc:654] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:49.269608 8877 raft_consensus.cc:357] T 03640a5124d2405faf4a26d0e22e1ffc P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:49.269876 8877 raft_consensus.cc:383] T 03640a5124d2405faf4a26d0e22e1ffc P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:49.269977 8877 raft_consensus.cc:738] T 03640a5124d2405faf4a26d0e22e1ffc P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Initialized, Role: FOLLOWER
I20250627 13:53:49.270242 8877 consensus_queue.cc:260] T 03640a5124d2405faf4a26d0e22e1ffc P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:49.270830 8877 ts_tablet_manager.cc:1428] T 03640a5124d2405faf4a26d0e22e1ffc P 9cf479bdcd0242738d2a77db4ad9f660: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:53:49.271144 8877 tablet_bootstrap.cc:492] T 87726824b05a4b968386f57d3a7a4aad P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap starting.
I20250627 13:53:49.271242 8880 tablet_bootstrap.cc:492] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e: No bootstrap required, opened a new log
I20250627 13:53:49.271389 8880 ts_tablet_manager.cc:1397] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250627 13:53:49.272137 8880 raft_consensus.cc:357] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:49.272311 8878 tablet_bootstrap.cc:654] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:49.272382 8880 raft_consensus.cc:383] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:49.272495 8880 raft_consensus.cc:738] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: FOLLOWER
I20250627 13:53:49.272747 8880 consensus_queue.cc:260] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:49.272822 8877 tablet_bootstrap.cc:654] T 87726824b05a4b968386f57d3a7a4aad P 9cf479bdcd0242738d2a77db4ad9f660: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:49.273209 8880 ts_tablet_manager.cc:1428] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250627 13:53:49.276226 8878 tablet_bootstrap.cc:492] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854: No bootstrap required, opened a new log
I20250627 13:53:49.276433 8878 ts_tablet_manager.cc:1397] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.001s
I20250627 13:53:49.277179 8878 raft_consensus.cc:357] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:49.277381 8878 raft_consensus.cc:383] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:49.277446 8878 raft_consensus.cc:738] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: FOLLOWER
I20250627 13:53:49.277736 8878 consensus_queue.cc:260] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:49.278270 8878 ts_tablet_manager.cc:1428] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:53:49.278556 8878 tablet_bootstrap.cc:492] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:53:49.278775 8877 tablet_bootstrap.cc:492] T 87726824b05a4b968386f57d3a7a4aad P 9cf479bdcd0242738d2a77db4ad9f660: No bootstrap required, opened a new log
I20250627 13:53:49.278928 8877 ts_tablet_manager.cc:1397] T 87726824b05a4b968386f57d3a7a4aad P 9cf479bdcd0242738d2a77db4ad9f660: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20250627 13:53:49.279464 8877 raft_consensus.cc:357] T 87726824b05a4b968386f57d3a7a4aad P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:49.279596 8877 raft_consensus.cc:383] T 87726824b05a4b968386f57d3a7a4aad P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:49.279686 8877 raft_consensus.cc:738] T 87726824b05a4b968386f57d3a7a4aad P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Initialized, Role: FOLLOWER
I20250627 13:53:49.279920 8877 consensus_queue.cc:260] T 87726824b05a4b968386f57d3a7a4aad P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:49.280031 8878 tablet_bootstrap.cc:654] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:49.280589 8877 ts_tablet_manager.cc:1428] T 87726824b05a4b968386f57d3a7a4aad P 9cf479bdcd0242738d2a77db4ad9f660: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250627 13:53:49.281916 8878 tablet_bootstrap.cc:492] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854: No bootstrap required, opened a new log
I20250627 13:53:49.282047 8878 ts_tablet_manager.cc:1397] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20250627 13:53:49.282687 8878 raft_consensus.cc:357] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:49.282814 8878 raft_consensus.cc:383] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:49.282908 8878 raft_consensus.cc:738] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: FOLLOWER
I20250627 13:53:49.283151 8878 consensus_queue.cc:260] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:49.283623 8878 ts_tablet_manager.cc:1428] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250627 13:53:49.344946 8857 raft_consensus.cc:491] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:49.345148 8857 raft_consensus.cc:513] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:49.346062 8857 leader_election.cc:290] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675), e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:53:49.346565 7933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "644985e3086d4614b6598a727cb60c1d" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" is_pre_election: true
I20250627 13:53:49.346858 7933 raft_consensus.cc:2466] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 0.
I20250627 13:53:49.347352 7998 leader_election.cc:304] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cf479bdcd0242738d2a77db4ad9f660, a41a89ae8ed743edb2b5cf4490d8c854; no voters:
I20250627 13:53:49.347658 8857 raft_consensus.cc:2802] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:49.347788 8857 raft_consensus.cc:491] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:49.347893 8857 raft_consensus.cc:3058] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:49.351075 8857 raft_consensus.cc:513] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:49.351827 8857 leader_election.cc:290] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 election: Requested vote from peers a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675), e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:53:49.352217 7933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "644985e3086d4614b6598a727cb60c1d" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
I20250627 13:53:49.352463 7933 raft_consensus.cc:3058] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:49.352912 8193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "644985e3086d4614b6598a727cb60c1d" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e" is_pre_election: true
I20250627 13:53:49.353032 8192 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "644985e3086d4614b6598a727cb60c1d" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e"
I20250627 13:53:49.353173 8193 raft_consensus.cc:2466] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 0.
I20250627 13:53:49.354764 7933 raft_consensus.cc:2466] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 1.
I20250627 13:53:49.355154 7998 leader_election.cc:304] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9cf479bdcd0242738d2a77db4ad9f660, a41a89ae8ed743edb2b5cf4490d8c854; no voters: e4c831df4da841929d13d6c783099b0e
I20250627 13:53:49.355427 8857 raft_consensus.cc:2802] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:49.355569 8857 raft_consensus.cc:695] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Becoming Leader. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Running, Role: LEADER
I20250627 13:53:49.355851 8857 consensus_queue.cc:237] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:49.358438 7659 catalog_manager.cc:5582] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:49.415488 8870 raft_consensus.cc:491] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:49.415685 8870 raft_consensus.cc:513] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:49.416565 8870 leader_election.cc:290] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:53:49.417068 8192 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03640a5124d2405faf4a26d0e22e1ffc" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e" is_pre_election: true
I20250627 13:53:49.417312 8192 raft_consensus.cc:2466] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a41a89ae8ed743edb2b5cf4490d8c854 in term 0.
I20250627 13:53:49.417794 7869 leader_election.cc:304] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a41a89ae8ed743edb2b5cf4490d8c854, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:49.418064 8870 raft_consensus.cc:2802] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:49.418193 8870 raft_consensus.cc:491] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:49.418303 8870 raft_consensus.cc:3058] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:49.420266 8870 raft_consensus.cc:513] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:49.420789 8870 leader_election.cc:290] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 1 election: Requested vote from peers e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:53:49.421207 8192 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03640a5124d2405faf4a26d0e22e1ffc" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e"
I20250627 13:53:49.421415 8192 raft_consensus.cc:3058] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:49.422812 8062 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03640a5124d2405faf4a26d0e22e1ffc" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660" is_pre_election: true
I20250627 13:53:49.422919 8063 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03640a5124d2405faf4a26d0e22e1ffc" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660"
I20250627 13:53:49.423041 8062 raft_consensus.cc:2466] T 03640a5124d2405faf4a26d0e22e1ffc P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a41a89ae8ed743edb2b5cf4490d8c854 in term 0.
I20250627 13:53:49.423832 8192 raft_consensus.cc:2466] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41a89ae8ed743edb2b5cf4490d8c854 in term 1.
I20250627 13:53:49.424193 7869 leader_election.cc:304] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a41a89ae8ed743edb2b5cf4490d8c854, e4c831df4da841929d13d6c783099b0e; no voters: 9cf479bdcd0242738d2a77db4ad9f660
I20250627 13:53:49.424463 8870 raft_consensus.cc:2802] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:49.424598 8870 raft_consensus.cc:695] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Becoming Leader. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Running, Role: LEADER
I20250627 13:53:49.424840 8870 consensus_queue.cc:237] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:49.427523 7659 catalog_manager.cc:5582] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: term changed from 0 to 1, leader changed from <none> to a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:49.726797 8796 raft_consensus.cc:491] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:49.726999 8796 raft_consensus.cc:513] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:49.728444 8796 leader_election.cc:290] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675), 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:53:49.734582 8063 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87726824b05a4b968386f57d3a7a4aad" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660" is_pre_election: true
I20250627 13:53:49.734813 8063 raft_consensus.cc:2466] T 87726824b05a4b968386f57d3a7a4aad P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 0.
I20250627 13:53:49.735265 7735 leader_election.cc:304] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, 9cf479bdcd0242738d2a77db4ad9f660; no voters:
I20250627 13:53:49.735633 8796 raft_consensus.cc:2802] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:49.735801 8796 raft_consensus.cc:491] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:49.735927 8796 raft_consensus.cc:3058] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:49.739094 8796 raft_consensus.cc:513] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:49.739670 8796 leader_election.cc:290] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 1 election: Requested vote from peers a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675), 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:53:49.740152 8063 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87726824b05a4b968386f57d3a7a4aad" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660"
I20250627 13:53:49.740337 8063 raft_consensus.cc:3058] T 87726824b05a4b968386f57d3a7a4aad P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:49.742798 8063 raft_consensus.cc:2466] T 87726824b05a4b968386f57d3a7a4aad P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 1.
I20250627 13:53:49.742962 7933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87726824b05a4b968386f57d3a7a4aad" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" is_pre_election: true
I20250627 13:53:49.743022 7932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87726824b05a4b968386f57d3a7a4aad" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
I20250627 13:53:49.743193 7933 raft_consensus.cc:2466] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 0.
I20250627 13:53:49.743252 7735 leader_election.cc:304] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, 9cf479bdcd0242738d2a77db4ad9f660; no voters:
I20250627 13:53:49.743556 8796 raft_consensus.cc:2802] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:49.743932 8796 raft_consensus.cc:695] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Becoming Leader. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Running, Role: LEADER
I20250627 13:53:49.744539 8796 consensus_queue.cc:237] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:49.747720 7659 catalog_manager.cc:5582] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83574c1bf8ea45adaff939158f93c096 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:49.817730 8192 raft_consensus.cc:3058] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:49.820374 8857 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:49.828502 8870 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:49.829010 8806 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:49.841537 8063 raft_consensus.cc:3058] T 03640a5124d2405faf4a26d0e22e1ffc P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:49.844671 8870 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250627 13:53:50.131776 8899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:53:50.132115 8899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:53:50.139699 8899 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20250627 13:53:50.196576 7932 raft_consensus.cc:3058] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:50.200343 8796 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:50.235404 8796 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 264.805 ms
time per row: 0.132402 ms
I20250627 13:53:50.550267 7661 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45288:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250627 13:53:50.558887 7781 tablet_service.cc:1468] Processing CreateTablet for tablet 40431e3bd7244904a4e65f59477bd21d (DEFAULT_TABLE table=rebalance_test_table_2 [id=6defd4a16efc47569b2193b71e4b2c4a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:50.559201 7913 tablet_service.cc:1468] Processing CreateTablet for tablet 40431e3bd7244904a4e65f59477bd21d (DEFAULT_TABLE table=rebalance_test_table_2 [id=6defd4a16efc47569b2193b71e4b2c4a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:50.559405 7781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40431e3bd7244904a4e65f59477bd21d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:50.559648 7913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40431e3bd7244904a4e65f59477bd21d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:50.560148 8041 tablet_service.cc:1468] Processing CreateTablet for tablet 5f176b66fdce4398be3492a5f3b4ea27 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6defd4a16efc47569b2193b71e4b2c4a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:50.560572 8041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f176b66fdce4398be3492a5f3b4ea27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:50.561234 8172 tablet_service.cc:1468] Processing CreateTablet for tablet 40431e3bd7244904a4e65f59477bd21d (DEFAULT_TABLE table=rebalance_test_table_2 [id=6defd4a16efc47569b2193b71e4b2c4a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:50.561676 8172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40431e3bd7244904a4e65f59477bd21d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:50.562122 8171 tablet_service.cc:1468] Processing CreateTablet for tablet da90822ff15e4e5da05114ef38d3970a (DEFAULT_TABLE table=rebalance_test_table_2 [id=6defd4a16efc47569b2193b71e4b2c4a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:50.562505 8171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da90822ff15e4e5da05114ef38d3970a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:50.563946 8173 tablet_service.cc:1468] Processing CreateTablet for tablet 5f176b66fdce4398be3492a5f3b4ea27 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6defd4a16efc47569b2193b71e4b2c4a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:50.564121 7912 tablet_service.cc:1468] Processing CreateTablet for tablet da90822ff15e4e5da05114ef38d3970a (DEFAULT_TABLE table=rebalance_test_table_2 [id=6defd4a16efc47569b2193b71e4b2c4a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:50.564311 8173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f176b66fdce4398be3492a5f3b4ea27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:50.564464 7912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da90822ff15e4e5da05114ef38d3970a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:50.559413 7782 tablet_service.cc:1468] Processing CreateTablet for tablet 5f176b66fdce4398be3492a5f3b4ea27 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6defd4a16efc47569b2193b71e4b2c4a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:50.570293 7782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f176b66fdce4398be3492a5f3b4ea27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:50.573372 8969 tablet_bootstrap.cc:492] T 40431e3bd7244904a4e65f59477bd21d P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
I20250627 13:53:50.574699 7780 tablet_service.cc:1468] Processing CreateTablet for tablet da90822ff15e4e5da05114ef38d3970a (DEFAULT_TABLE table=rebalance_test_table_2 [id=6defd4a16efc47569b2193b71e4b2c4a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:50.575078 7780 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da90822ff15e4e5da05114ef38d3970a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:50.575232 8969 tablet_bootstrap.cc:654] T 40431e3bd7244904a4e65f59477bd21d P e4c831df4da841929d13d6c783099b0e: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:50.586190 8968 tablet_bootstrap.cc:492] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:53:50.586894 8971 tablet_bootstrap.cc:492] T 5f176b66fdce4398be3492a5f3b4ea27 P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap starting.
I20250627 13:53:50.587486 8969 tablet_bootstrap.cc:492] T 40431e3bd7244904a4e65f59477bd21d P e4c831df4da841929d13d6c783099b0e: No bootstrap required, opened a new log
I20250627 13:53:50.587725 8969 ts_tablet_manager.cc:1397] T 40431e3bd7244904a4e65f59477bd21d P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.007s
I20250627 13:53:50.588141 8968 tablet_bootstrap.cc:654] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:50.588786 8972 tablet_bootstrap.cc:492] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:53:50.588740 8969 raft_consensus.cc:357] T 40431e3bd7244904a4e65f59477bd21d P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:50.589047 8969 raft_consensus.cc:383] T 40431e3bd7244904a4e65f59477bd21d P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:50.589215 8969 raft_consensus.cc:738] T 40431e3bd7244904a4e65f59477bd21d P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: FOLLOWER
I20250627 13:53:50.589316 8971 tablet_bootstrap.cc:654] T 5f176b66fdce4398be3492a5f3b4ea27 P 9cf479bdcd0242738d2a77db4ad9f660: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:50.589583 8969 consensus_queue.cc:260] T 40431e3bd7244904a4e65f59477bd21d P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:50.591285 8969 ts_tablet_manager.cc:1428] T 40431e3bd7244904a4e65f59477bd21d P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250627 13:53:50.591744 8969 tablet_bootstrap.cc:492] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
I20250627 13:53:50.594159 8969 tablet_bootstrap.cc:654] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:50.597584 8968 tablet_bootstrap.cc:492] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096: No bootstrap required, opened a new log
I20250627 13:53:50.597774 8968 ts_tablet_manager.cc:1397] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.010s
I20250627 13:53:50.598654 8972 tablet_bootstrap.cc:654] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:50.598630 8968 raft_consensus.cc:357] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:50.598819 8968 raft_consensus.cc:383] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:50.598910 8968 raft_consensus.cc:738] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: FOLLOWER
I20250627 13:53:50.599154 8968 consensus_queue.cc:260] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:50.599701 8969 tablet_bootstrap.cc:492] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e: No bootstrap required, opened a new log
I20250627 13:53:50.599783 8968 ts_tablet_manager.cc:1428] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:53:50.599875 8969 ts_tablet_manager.cc:1397] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20250627 13:53:50.600069 8968 tablet_bootstrap.cc:492] T da90822ff15e4e5da05114ef38d3970a P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:53:50.600723 8969 raft_consensus.cc:357] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:50.600901 8969 raft_consensus.cc:383] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:50.600970 8969 raft_consensus.cc:738] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: FOLLOWER
I20250627 13:53:50.601222 8969 consensus_queue.cc:260] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:50.601742 8969 ts_tablet_manager.cc:1428] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250627 13:53:50.602025 8969 tablet_bootstrap.cc:492] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
I20250627 13:53:50.604244 8968 tablet_bootstrap.cc:654] T da90822ff15e4e5da05114ef38d3970a P 83574c1bf8ea45adaff939158f93c096: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:50.604492 8969 tablet_bootstrap.cc:654] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:50.606659 8971 tablet_bootstrap.cc:492] T 5f176b66fdce4398be3492a5f3b4ea27 P 9cf479bdcd0242738d2a77db4ad9f660: No bootstrap required, opened a new log
I20250627 13:53:50.606810 8971 ts_tablet_manager.cc:1397] T 5f176b66fdce4398be3492a5f3b4ea27 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.000s
I20250627 13:53:50.607218 8968 tablet_bootstrap.cc:492] T da90822ff15e4e5da05114ef38d3970a P 83574c1bf8ea45adaff939158f93c096: No bootstrap required, opened a new log
I20250627 13:53:50.607403 8969 tablet_bootstrap.cc:492] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e: No bootstrap required, opened a new log
I20250627 13:53:50.607450 8968 ts_tablet_manager.cc:1397] T da90822ff15e4e5da05114ef38d3970a P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20250627 13:53:50.607534 8969 ts_tablet_manager.cc:1397] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250627 13:53:50.607705 8971 raft_consensus.cc:357] T 5f176b66fdce4398be3492a5f3b4ea27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:50.607888 8971 raft_consensus.cc:383] T 5f176b66fdce4398be3492a5f3b4ea27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:50.607970 8971 raft_consensus.cc:738] T 5f176b66fdce4398be3492a5f3b4ea27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Initialized, Role: FOLLOWER
I20250627 13:53:50.608172 8971 consensus_queue.cc:260] T 5f176b66fdce4398be3492a5f3b4ea27 P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:50.608248 8969 raft_consensus.cc:357] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:50.608387 8969 raft_consensus.cc:383] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:50.608465 8969 raft_consensus.cc:738] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: FOLLOWER
I20250627 13:53:50.608387 8968 raft_consensus.cc:357] T da90822ff15e4e5da05114ef38d3970a P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:50.608633 8968 raft_consensus.cc:383] T da90822ff15e4e5da05114ef38d3970a P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:50.608732 8971 ts_tablet_manager.cc:1428] T 5f176b66fdce4398be3492a5f3b4ea27 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:53:50.608716 8968 raft_consensus.cc:738] T da90822ff15e4e5da05114ef38d3970a P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: FOLLOWER
I20250627 13:53:50.608686 8969 consensus_queue.cc:260] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:50.609150 8969 ts_tablet_manager.cc:1428] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250627 13:53:50.609501 8968 consensus_queue.cc:260] T da90822ff15e4e5da05114ef38d3970a P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:50.610605 8968 ts_tablet_manager.cc:1428] T da90822ff15e4e5da05114ef38d3970a P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20250627 13:53:50.610833 8972 tablet_bootstrap.cc:492] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854: No bootstrap required, opened a new log
I20250627 13:53:50.610936 8968 tablet_bootstrap.cc:492] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:53:50.611006 8972 ts_tablet_manager.cc:1397] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.003s
I20250627 13:53:50.611815 8972 raft_consensus.cc:357] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:50.612147 8972 raft_consensus.cc:383] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:50.612294 8972 raft_consensus.cc:738] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: FOLLOWER
I20250627 13:53:50.612574 8972 consensus_queue.cc:260] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:50.612901 8968 tablet_bootstrap.cc:654] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:50.613127 8972 ts_tablet_manager.cc:1428] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:53:50.614056 8972 tablet_bootstrap.cc:492] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:53:50.615110 8968 tablet_bootstrap.cc:492] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096: No bootstrap required, opened a new log
I20250627 13:53:50.615336 8968 ts_tablet_manager.cc:1397] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20250627 13:53:50.615728 8972 tablet_bootstrap.cc:654] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:50.616046 8968 raft_consensus.cc:357] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:50.616287 8968 raft_consensus.cc:383] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:50.616433 8968 raft_consensus.cc:738] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: FOLLOWER
I20250627 13:53:50.616695 8968 consensus_queue.cc:260] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:50.617153 8968 ts_tablet_manager.cc:1428] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250627 13:53:50.618712 8972 tablet_bootstrap.cc:492] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854: No bootstrap required, opened a new log
I20250627 13:53:50.618860 8972 ts_tablet_manager.cc:1397] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20250627 13:53:50.619566 8972 raft_consensus.cc:357] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:50.619709 8972 raft_consensus.cc:383] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:50.619783 8972 raft_consensus.cc:738] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: FOLLOWER
I20250627 13:53:50.620016 8972 consensus_queue.cc:260] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:50.620505 8972 ts_tablet_manager.cc:1428] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250627 13:53:50.687001 8952 raft_consensus.cc:491] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:50.687176 8952 raft_consensus.cc:513] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:50.688037 8952 leader_election.cc:290] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675), e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:53:50.688428 7932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40431e3bd7244904a4e65f59477bd21d" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" is_pre_election: true
I20250627 13:53:50.688643 7932 raft_consensus.cc:2466] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 0.
I20250627 13:53:50.689122 7737 leader_election.cc:304] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, a41a89ae8ed743edb2b5cf4490d8c854; no voters:
I20250627 13:53:50.689390 8952 raft_consensus.cc:2802] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:50.689502 8952 raft_consensus.cc:491] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:50.689612 8952 raft_consensus.cc:3058] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:50.692044 8952 raft_consensus.cc:513] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:50.692801 8952 leader_election.cc:290] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 1 election: Requested vote from peers a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675), e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:53:50.693274 7932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40431e3bd7244904a4e65f59477bd21d" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
I20250627 13:53:50.693468 7932 raft_consensus.cc:3058] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:50.695575 8193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40431e3bd7244904a4e65f59477bd21d" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e"
I20250627 13:53:50.695544 8192 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40431e3bd7244904a4e65f59477bd21d" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e" is_pre_election: true
I20250627 13:53:50.695786 8193 raft_consensus.cc:3058] T 40431e3bd7244904a4e65f59477bd21d P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:50.695874 7932 raft_consensus.cc:2466] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 1.
I20250627 13:53:50.696357 7737 leader_election.cc:304] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, a41a89ae8ed743edb2b5cf4490d8c854; no voters:
I20250627 13:53:50.696698 8952 raft_consensus.cc:2802] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:50.696847 8952 raft_consensus.cc:695] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Becoming Leader. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Running, Role: LEADER
I20250627 13:53:50.697202 8952 consensus_queue.cc:237] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:50.698103 8193 raft_consensus.cc:2466] T 40431e3bd7244904a4e65f59477bd21d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 1.
I20250627 13:53:50.700074 7661 catalog_manager.cc:5582] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83574c1bf8ea45adaff939158f93c096 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:50.894368 8855 raft_consensus.cc:491] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:50.894555 8855 raft_consensus.cc:513] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:50.895228 8855 leader_election.cc:290] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467), 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:53:50.895740 7801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f176b66fdce4398be3492a5f3b4ea27" candidate_uuid: "e4c831df4da841929d13d6c783099b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096" is_pre_election: true
I20250627 13:53:50.895787 8062 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f176b66fdce4398be3492a5f3b4ea27" candidate_uuid: "e4c831df4da841929d13d6c783099b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660" is_pre_election: true
I20250627 13:53:50.895983 8062 raft_consensus.cc:2466] T 5f176b66fdce4398be3492a5f3b4ea27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4c831df4da841929d13d6c783099b0e in term 0.
I20250627 13:53:50.895983 7801 raft_consensus.cc:2466] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4c831df4da841929d13d6c783099b0e in term 0.
I20250627 13:53:50.896471 8126 leader_election.cc:304] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:50.896733 8855 raft_consensus.cc:2802] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:50.896867 8855 raft_consensus.cc:491] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:50.896955 8855 raft_consensus.cc:3058] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:50.898869 8855 raft_consensus.cc:513] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:50.899369 8855 leader_election.cc:290] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [CANDIDATE]: Term 1 election: Requested vote from peers 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467), 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:53:50.899818 7801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f176b66fdce4398be3492a5f3b4ea27" candidate_uuid: "e4c831df4da841929d13d6c783099b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096"
I20250627 13:53:50.899928 8062 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f176b66fdce4398be3492a5f3b4ea27" candidate_uuid: "e4c831df4da841929d13d6c783099b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660"
I20250627 13:53:50.900002 7801 raft_consensus.cc:3058] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:50.900110 8062 raft_consensus.cc:3058] T 5f176b66fdce4398be3492a5f3b4ea27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:50.902885 8062 raft_consensus.cc:2466] T 5f176b66fdce4398be3492a5f3b4ea27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c831df4da841929d13d6c783099b0e in term 1.
I20250627 13:53:50.902892 7801 raft_consensus.cc:2466] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c831df4da841929d13d6c783099b0e in term 1.
I20250627 13:53:50.903309 8126 leader_election.cc:304] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:50.903551 8855 raft_consensus.cc:2802] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:50.903930 8855 raft_consensus.cc:695] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Becoming Leader. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Running, Role: LEADER
I20250627 13:53:50.904258 8855 consensus_queue.cc:237] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:50.904806 8984 raft_consensus.cc:491] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:50.905095 8984 raft_consensus.cc:513] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:50.906113 8984 leader_election.cc:290] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675), 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:53:50.907372 7801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da90822ff15e4e5da05114ef38d3970a" candidate_uuid: "e4c831df4da841929d13d6c783099b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096" is_pre_election: true
I20250627 13:53:50.907641 7801 raft_consensus.cc:2466] T da90822ff15e4e5da05114ef38d3970a P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4c831df4da841929d13d6c783099b0e in term 0.
I20250627 13:53:50.908268 8126 leader_election.cc:304] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:50.908634 8984 raft_consensus.cc:2802] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:50.908866 8984 raft_consensus.cc:491] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:50.908995 8984 raft_consensus.cc:3058] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:50.911257 8984 raft_consensus.cc:513] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:50.911676 8984 leader_election.cc:290] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [CANDIDATE]: Term 1 election: Requested vote from peers a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675), 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:53:50.912294 7801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da90822ff15e4e5da05114ef38d3970a" candidate_uuid: "e4c831df4da841929d13d6c783099b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096"
I20250627 13:53:50.912513 7801 raft_consensus.cc:3058] T da90822ff15e4e5da05114ef38d3970a P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:50.916116 7801 raft_consensus.cc:2466] T da90822ff15e4e5da05114ef38d3970a P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c831df4da841929d13d6c783099b0e in term 1.
I20250627 13:53:50.916579 8126 leader_election.cc:304] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:50.917498 8984 raft_consensus.cc:2802] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:50.925036 8984 raft_consensus.cc:695] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Becoming Leader. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Running, Role: LEADER
I20250627 13:53:50.925403 8984 consensus_queue.cc:237] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:50.927644 7661 catalog_manager.cc:5582] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e reported cstate change: term changed from 0 to 1, leader changed from <none> to e4c831df4da841929d13d6c783099b0e (127.0.103.68). New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:50.931056 7932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da90822ff15e4e5da05114ef38d3970a" candidate_uuid: "e4c831df4da841929d13d6c783099b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" is_pre_election: true
I20250627 13:53:50.931329 7932 raft_consensus.cc:2466] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4c831df4da841929d13d6c783099b0e in term 0.
I20250627 13:53:50.931686 7933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da90822ff15e4e5da05114ef38d3970a" candidate_uuid: "e4c831df4da841929d13d6c783099b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
I20250627 13:53:50.931851 7933 raft_consensus.cc:3058] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:50.935041 7933 raft_consensus.cc:2466] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c831df4da841929d13d6c783099b0e in term 1.
I20250627 13:53:50.935611 7661 catalog_manager.cc:5582] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e reported cstate change: term changed from 0 to 1, leader changed from <none> to e4c831df4da841929d13d6c783099b0e (127.0.103.68). New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:51.135111 8952 consensus_queue.cc:1035] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:51.143735 8952 consensus_queue.cc:1035] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:51.301085 8984 consensus_queue.cc:1035] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:51.307060 8855 consensus_queue.cc:1035] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250627 13:53:51.360184 8990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:53:51.360617 8990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:53:51.368618 8990 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250627 13:53:51.385555 8855 consensus_queue.cc:1035] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:51.390789 8855 consensus_queue.cc:1035] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 241.608 ms
time per row: 0.120804 ms
I20250627 13:53:51.755775 7659 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45288:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250627 13:53:51.764782 7781 tablet_service.cc:1468] Processing CreateTablet for tablet 1dc569a7b12d40058d073f7b81b2601c (DEFAULT_TABLE table=rebalance_test_table_3 [id=ec6a9ed6ab4d416988b2abe2f8d29c55]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:51.764768 8041 tablet_service.cc:1468] Processing CreateTablet for tablet 1dc569a7b12d40058d073f7b81b2601c (DEFAULT_TABLE table=rebalance_test_table_3 [id=ec6a9ed6ab4d416988b2abe2f8d29c55]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:51.765220 8041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dc569a7b12d40058d073f7b81b2601c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:51.765220 7781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dc569a7b12d40058d073f7b81b2601c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:51.765957 7912 tablet_service.cc:1468] Processing CreateTablet for tablet 1dc569a7b12d40058d073f7b81b2601c (DEFAULT_TABLE table=rebalance_test_table_3 [id=ec6a9ed6ab4d416988b2abe2f8d29c55]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:51.766360 7912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dc569a7b12d40058d073f7b81b2601c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:51.766614 8043 tablet_service.cc:1468] Processing CreateTablet for tablet 5184c3f8e465472e9a98c14f8e703415 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ec6a9ed6ab4d416988b2abe2f8d29c55]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:51.766991 8043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5184c3f8e465472e9a98c14f8e703415. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:51.767506 8042 tablet_service.cc:1468] Processing CreateTablet for tablet 6f5e370b77cc42fdb0b886ea19af06c3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ec6a9ed6ab4d416988b2abe2f8d29c55]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:51.768497 7782 tablet_service.cc:1468] Processing CreateTablet for tablet 5184c3f8e465472e9a98c14f8e703415 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ec6a9ed6ab4d416988b2abe2f8d29c55]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:51.768874 7782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5184c3f8e465472e9a98c14f8e703415. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:51.770090 7913 tablet_service.cc:1468] Processing CreateTablet for tablet 6f5e370b77cc42fdb0b886ea19af06c3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ec6a9ed6ab4d416988b2abe2f8d29c55]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:51.770478 7913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f5e370b77cc42fdb0b886ea19af06c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:51.770936 8042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f5e370b77cc42fdb0b886ea19af06c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:51.772298 8171 tablet_service.cc:1468] Processing CreateTablet for tablet 5184c3f8e465472e9a98c14f8e703415 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ec6a9ed6ab4d416988b2abe2f8d29c55]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:51.772861 8171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5184c3f8e465472e9a98c14f8e703415. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:51.778087 8173 tablet_service.cc:1468] Processing CreateTablet for tablet 6f5e370b77cc42fdb0b886ea19af06c3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ec6a9ed6ab4d416988b2abe2f8d29c55]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:51.778532 8173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f5e370b77cc42fdb0b886ea19af06c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:51.783586 9056 tablet_bootstrap.cc:492] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:53:51.788266 9056 tablet_bootstrap.cc:654] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:51.788355 9055 tablet_bootstrap.cc:492] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap starting.
I20250627 13:53:51.790303 9055 tablet_bootstrap.cc:654] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:51.802549 9056 tablet_bootstrap.cc:492] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854: No bootstrap required, opened a new log
I20250627 13:53:51.802814 9054 tablet_bootstrap.cc:492] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:53:51.803215 9056 ts_tablet_manager.cc:1397] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.010s
I20250627 13:53:51.804368 9056 raft_consensus.cc:357] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:51.804767 9056 raft_consensus.cc:383] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:51.804927 9056 raft_consensus.cc:738] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: FOLLOWER
I20250627 13:53:51.805308 9054 tablet_bootstrap.cc:654] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:51.805320 9056 consensus_queue.cc:260] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:51.806950 9056 ts_tablet_manager.cc:1428] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250627 13:53:51.806946 9059 tablet_bootstrap.cc:492] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
I20250627 13:53:51.807271 9056 tablet_bootstrap.cc:492] T 6f5e370b77cc42fdb0b886ea19af06c3 P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:53:51.809021 9056 tablet_bootstrap.cc:654] T 6f5e370b77cc42fdb0b886ea19af06c3 P a41a89ae8ed743edb2b5cf4490d8c854: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:51.809358 9059 tablet_bootstrap.cc:654] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:51.810583 9055 tablet_bootstrap.cc:492] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660: No bootstrap required, opened a new log
I20250627 13:53:51.810771 9055 ts_tablet_manager.cc:1397] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.011s
I20250627 13:53:51.811574 9055 raft_consensus.cc:357] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:51.811841 9055 raft_consensus.cc:383] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:51.812749 9055 raft_consensus.cc:738] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Initialized, Role: FOLLOWER
I20250627 13:53:51.812930 9056 tablet_bootstrap.cc:492] T 6f5e370b77cc42fdb0b886ea19af06c3 P a41a89ae8ed743edb2b5cf4490d8c854: No bootstrap required, opened a new log
I20250627 13:53:51.813138 9056 ts_tablet_manager.cc:1397] T 6f5e370b77cc42fdb0b886ea19af06c3 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20250627 13:53:51.813990 9056 raft_consensus.cc:357] T 6f5e370b77cc42fdb0b886ea19af06c3 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:51.814239 9056 raft_consensus.cc:383] T 6f5e370b77cc42fdb0b886ea19af06c3 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:51.813087 9055 consensus_queue.cc:260] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:51.814355 9056 raft_consensus.cc:738] T 6f5e370b77cc42fdb0b886ea19af06c3 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: FOLLOWER
I20250627 13:53:51.814934 9055 ts_tablet_manager.cc:1428] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250627 13:53:51.815243 9055 tablet_bootstrap.cc:492] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap starting.
I20250627 13:53:51.815735 9056 consensus_queue.cc:260] T 6f5e370b77cc42fdb0b886ea19af06c3 P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:51.817173 9055 tablet_bootstrap.cc:654] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:51.817776 9056 ts_tablet_manager.cc:1428] T 6f5e370b77cc42fdb0b886ea19af06c3 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250627 13:53:51.825150 9059 tablet_bootstrap.cc:492] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e: No bootstrap required, opened a new log
I20250627 13:53:51.825313 9059 ts_tablet_manager.cc:1397] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.005s
I20250627 13:53:51.825707 9054 tablet_bootstrap.cc:492] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096: No bootstrap required, opened a new log
I20250627 13:53:51.825863 9054 ts_tablet_manager.cc:1397] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.000s
I20250627 13:53:51.826143 9055 tablet_bootstrap.cc:492] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660: No bootstrap required, opened a new log
I20250627 13:53:51.826301 9055 ts_tablet_manager.cc:1397] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20250627 13:53:51.826195 9059 raft_consensus.cc:357] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:51.826390 9059 raft_consensus.cc:383] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:51.826485 9059 raft_consensus.cc:738] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: FOLLOWER
I20250627 13:53:51.826723 9054 raft_consensus.cc:357] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:51.826704 9059 consensus_queue.cc:260] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:51.826905 9054 raft_consensus.cc:383] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:51.826946 9055 raft_consensus.cc:357] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:51.827137 9055 raft_consensus.cc:383] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:51.827206 9055 raft_consensus.cc:738] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Initialized, Role: FOLLOWER
I20250627 13:53:51.827332 9059 ts_tablet_manager.cc:1428] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:53:51.827427 9055 consensus_queue.cc:260] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:51.827651 9059 tablet_bootstrap.cc:492] T 5184c3f8e465472e9a98c14f8e703415 P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
I20250627 13:53:51.827643 9054 raft_consensus.cc:738] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: FOLLOWER
I20250627 13:53:51.827903 9054 consensus_queue.cc:260] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:51.828436 9055 ts_tablet_manager.cc:1428] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:53:51.828753 9055 tablet_bootstrap.cc:492] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap starting.
I20250627 13:53:51.830935 9054 ts_tablet_manager.cc:1428] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250627 13:53:51.831087 9055 tablet_bootstrap.cc:654] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:51.832466 9059 tablet_bootstrap.cc:654] T 5184c3f8e465472e9a98c14f8e703415 P e4c831df4da841929d13d6c783099b0e: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:51.834720 9054 tablet_bootstrap.cc:492] T 5184c3f8e465472e9a98c14f8e703415 P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:53:51.836875 9054 tablet_bootstrap.cc:654] T 5184c3f8e465472e9a98c14f8e703415 P 83574c1bf8ea45adaff939158f93c096: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:51.840528 9054 tablet_bootstrap.cc:492] T 5184c3f8e465472e9a98c14f8e703415 P 83574c1bf8ea45adaff939158f93c096: No bootstrap required, opened a new log
I20250627 13:53:51.840699 9054 ts_tablet_manager.cc:1397] T 5184c3f8e465472e9a98c14f8e703415 P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20250627 13:53:51.840819 9055 tablet_bootstrap.cc:492] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660: No bootstrap required, opened a new log
I20250627 13:53:51.840973 9055 ts_tablet_manager.cc:1397] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.007s
I20250627 13:53:51.841226 9059 tablet_bootstrap.cc:492] T 5184c3f8e465472e9a98c14f8e703415 P e4c831df4da841929d13d6c783099b0e: No bootstrap required, opened a new log
I20250627 13:53:51.841365 9059 ts_tablet_manager.cc:1397] T 5184c3f8e465472e9a98c14f8e703415 P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20250627 13:53:51.841462 9054 raft_consensus.cc:357] T 5184c3f8e465472e9a98c14f8e703415 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:51.841595 9055 raft_consensus.cc:357] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:51.841706 9054 raft_consensus.cc:383] T 5184c3f8e465472e9a98c14f8e703415 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:51.841789 9055 raft_consensus.cc:383] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:51.841779 9054 raft_consensus.cc:738] T 5184c3f8e465472e9a98c14f8e703415 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: FOLLOWER
I20250627 13:53:51.841874 9055 raft_consensus.cc:738] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Initialized, Role: FOLLOWER
I20250627 13:53:51.842028 9054 consensus_queue.cc:260] T 5184c3f8e465472e9a98c14f8e703415 P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:51.842052 9059 raft_consensus.cc:357] T 5184c3f8e465472e9a98c14f8e703415 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:51.842129 9055 consensus_queue.cc:260] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:51.842260 9059 raft_consensus.cc:383] T 5184c3f8e465472e9a98c14f8e703415 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:51.842327 9059 raft_consensus.cc:738] T 5184c3f8e465472e9a98c14f8e703415 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: FOLLOWER
I20250627 13:53:51.842607 9059 consensus_queue.cc:260] T 5184c3f8e465472e9a98c14f8e703415 P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:51.842794 9055 ts_tablet_manager.cc:1428] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:53:51.843071 9054 ts_tablet_manager.cc:1428] T 5184c3f8e465472e9a98c14f8e703415 P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:53:51.843464 9059 ts_tablet_manager.cc:1428] T 5184c3f8e465472e9a98c14f8e703415 P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:53:51.970237 8806 raft_consensus.cc:491] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:51.970456 8806 raft_consensus.cc:513] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:51.971114 8806 leader_election.cc:290] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467), e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:53:51.971714 8193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5184c3f8e465472e9a98c14f8e703415" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e" is_pre_election: true
I20250627 13:53:51.971719 7801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5184c3f8e465472e9a98c14f8e703415" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096" is_pre_election: true
I20250627 13:53:51.971922 8193 raft_consensus.cc:2466] T 5184c3f8e465472e9a98c14f8e703415 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 0.
I20250627 13:53:51.971949 7801 raft_consensus.cc:2466] T 5184c3f8e465472e9a98c14f8e703415 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 0.
I20250627 13:53:51.972390 7999 leader_election.cc:304] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cf479bdcd0242738d2a77db4ad9f660, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:51.972719 8806 raft_consensus.cc:2802] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:51.972816 8806 raft_consensus.cc:491] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:51.972908 8806 raft_consensus.cc:3058] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:51.974799 8806 raft_consensus.cc:513] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:51.975327 8806 leader_election.cc:290] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 election: Requested vote from peers 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467), e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:53:51.975659 7801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5184c3f8e465472e9a98c14f8e703415" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096"
I20250627 13:53:51.975791 8193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5184c3f8e465472e9a98c14f8e703415" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e"
I20250627 13:53:51.975880 7801 raft_consensus.cc:3058] T 5184c3f8e465472e9a98c14f8e703415 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:51.975989 8193 raft_consensus.cc:3058] T 5184c3f8e465472e9a98c14f8e703415 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:51.977907 8193 raft_consensus.cc:2466] T 5184c3f8e465472e9a98c14f8e703415 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 1.
I20250627 13:53:51.977934 7801 raft_consensus.cc:2466] T 5184c3f8e465472e9a98c14f8e703415 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 1.
I20250627 13:53:51.978281 7999 leader_election.cc:304] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cf479bdcd0242738d2a77db4ad9f660, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:51.978511 8806 raft_consensus.cc:2802] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:51.978878 8806 raft_consensus.cc:695] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Becoming Leader. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Running, Role: LEADER
I20250627 13:53:51.979140 8806 consensus_queue.cc:237] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:51.981418 7661 catalog_manager.cc:5582] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:52.047585 8952 raft_consensus.cc:491] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:52.047813 8952 raft_consensus.cc:513] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:52.048453 8952 leader_election.cc:290] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933), a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:53:52.048924 8061 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dc569a7b12d40058d073f7b81b2601c" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660" is_pre_election: true
I20250627 13:53:52.048959 7933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dc569a7b12d40058d073f7b81b2601c" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" is_pre_election: true
I20250627 13:53:52.049130 8061 raft_consensus.cc:2466] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 0.
I20250627 13:53:52.049168 7933 raft_consensus.cc:2466] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 0.
I20250627 13:53:52.049566 7737 leader_election.cc:304] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, a41a89ae8ed743edb2b5cf4490d8c854; no voters:
I20250627 13:53:52.049887 8952 raft_consensus.cc:2802] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:52.049968 8952 raft_consensus.cc:491] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:52.050069 8952 raft_consensus.cc:3058] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:52.051940 8952 raft_consensus.cc:513] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:52.052590 8952 leader_election.cc:290] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 1 election: Requested vote from peers 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933), a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:53:52.052873 8061 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dc569a7b12d40058d073f7b81b2601c" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660"
I20250627 13:53:52.053036 8061 raft_consensus.cc:3058] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:52.053138 7933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dc569a7b12d40058d073f7b81b2601c" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
I20250627 13:53:52.053334 7933 raft_consensus.cc:3058] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:52.055011 8061 raft_consensus.cc:2466] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 1.
I20250627 13:53:52.055222 7933 raft_consensus.cc:2466] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 1.
I20250627 13:53:52.055425 7735 leader_election.cc:304] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, 9cf479bdcd0242738d2a77db4ad9f660; no voters:
I20250627 13:53:52.055673 8952 raft_consensus.cc:2802] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:52.055825 8952 raft_consensus.cc:695] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Becoming Leader. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Running, Role: LEADER
I20250627 13:53:52.056089 8952 consensus_queue.cc:237] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:52.058492 7661 catalog_manager.cc:5582] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83574c1bf8ea45adaff939158f93c096 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:52.069311 8855 raft_consensus.cc:491] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:52.069486 8855 raft_consensus.cc:513] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:52.070086 8855 leader_election.cc:290] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933), a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:53:52.070497 8061 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f5e370b77cc42fdb0b886ea19af06c3" candidate_uuid: "e4c831df4da841929d13d6c783099b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660" is_pre_election: true
I20250627 13:53:52.070544 7933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f5e370b77cc42fdb0b886ea19af06c3" candidate_uuid: "e4c831df4da841929d13d6c783099b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" is_pre_election: true
I20250627 13:53:52.070694 8061 raft_consensus.cc:2466] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4c831df4da841929d13d6c783099b0e in term 0.
I20250627 13:53:52.070712 7933 raft_consensus.cc:2466] T 6f5e370b77cc42fdb0b886ea19af06c3 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4c831df4da841929d13d6c783099b0e in term 0.
I20250627 13:53:52.071064 8128 leader_election.cc:304] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a41a89ae8ed743edb2b5cf4490d8c854, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:52.071343 8855 raft_consensus.cc:2802] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:52.071455 8855 raft_consensus.cc:491] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:52.071549 8855 raft_consensus.cc:3058] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:52.073660 8855 raft_consensus.cc:513] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:52.074234 8855 leader_election.cc:290] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [CANDIDATE]: Term 1 election: Requested vote from peers 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933), a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:53:52.074692 8061 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f5e370b77cc42fdb0b886ea19af06c3" candidate_uuid: "e4c831df4da841929d13d6c783099b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660"
I20250627 13:53:52.074864 7933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f5e370b77cc42fdb0b886ea19af06c3" candidate_uuid: "e4c831df4da841929d13d6c783099b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
I20250627 13:53:52.074924 8061 raft_consensus.cc:3058] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:52.075043 7933 raft_consensus.cc:3058] T 6f5e370b77cc42fdb0b886ea19af06c3 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:52.077916 8061 raft_consensus.cc:2466] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c831df4da841929d13d6c783099b0e in term 1.
I20250627 13:53:52.078106 7933 raft_consensus.cc:2466] T 6f5e370b77cc42fdb0b886ea19af06c3 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c831df4da841929d13d6c783099b0e in term 1.
I20250627 13:53:52.078431 8126 leader_election.cc:304] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cf479bdcd0242738d2a77db4ad9f660, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:52.078680 8855 raft_consensus.cc:2802] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:52.078802 8855 raft_consensus.cc:695] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Becoming Leader. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Running, Role: LEADER
I20250627 13:53:52.079044 8855 consensus_queue.cc:237] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:52.081338 7661 catalog_manager.cc:5582] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e reported cstate change: term changed from 0 to 1, leader changed from <none> to e4c831df4da841929d13d6c783099b0e (127.0.103.68). New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } }
I20250627 13:53:52.407590 8806 consensus_queue.cc:1035] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:52.426059 9067 consensus_queue.cc:1035] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:52.477861 8952 consensus_queue.cc:1035] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:52.482113 8947 consensus_queue.cc:1035] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250627 13:53:52.506884 9071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:53:52.507261 9071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:53:52.514951 9071 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20250627 13:53:52.550438 8855 consensus_queue.cc:1035] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:52.554476 8855 consensus_queue.cc:1035] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 241.922 ms
time per row: 0.120961 ms
I20250627 13:53:52.909286 7661 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45288:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250627 13:53:52.918094 7781 tablet_service.cc:1468] Processing CreateTablet for tablet cff879c8e57d42c28440b3bd310f1ba9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=4d92d465970b4ec4b5eace263b25362a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:52.918365 7912 tablet_service.cc:1468] Processing CreateTablet for tablet cff879c8e57d42c28440b3bd310f1ba9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=4d92d465970b4ec4b5eace263b25362a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:52.918567 7781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cff879c8e57d42c28440b3bd310f1ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:52.918741 7912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cff879c8e57d42c28440b3bd310f1ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:52.919706 7913 tablet_service.cc:1468] Processing CreateTablet for tablet 897206e017ff4305a8cc9092e30f46b6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=4d92d465970b4ec4b5eace263b25362a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:52.920075 7913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 897206e017ff4305a8cc9092e30f46b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:52.926260 8041 tablet_service.cc:1468] Processing CreateTablet for tablet cff879c8e57d42c28440b3bd310f1ba9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=4d92d465970b4ec4b5eace263b25362a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:52.926389 8042 tablet_service.cc:1468] Processing CreateTablet for tablet 9196497cb11f4ccbbd62695253d4c8c9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=4d92d465970b4ec4b5eace263b25362a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:52.926723 8041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cff879c8e57d42c28440b3bd310f1ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:52.926950 8042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9196497cb11f4ccbbd62695253d4c8c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:52.928289 7782 tablet_service.cc:1468] Processing CreateTablet for tablet 897206e017ff4305a8cc9092e30f46b6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=4d92d465970b4ec4b5eace263b25362a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:52.928658 7782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 897206e017ff4305a8cc9092e30f46b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:52.930265 8173 tablet_service.cc:1468] Processing CreateTablet for tablet 897206e017ff4305a8cc9092e30f46b6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=4d92d465970b4ec4b5eace263b25362a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:52.930701 8173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 897206e017ff4305a8cc9092e30f46b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:52.930778 9137 tablet_bootstrap.cc:492] T cff879c8e57d42c28440b3bd310f1ba9 P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:53:52.932648 9137 tablet_bootstrap.cc:654] T cff879c8e57d42c28440b3bd310f1ba9 P 83574c1bf8ea45adaff939158f93c096: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:52.933993 7780 tablet_service.cc:1468] Processing CreateTablet for tablet 9196497cb11f4ccbbd62695253d4c8c9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=4d92d465970b4ec4b5eace263b25362a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:52.933995 8171 tablet_service.cc:1468] Processing CreateTablet for tablet 9196497cb11f4ccbbd62695253d4c8c9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=4d92d465970b4ec4b5eace263b25362a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:52.934326 7780 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9196497cb11f4ccbbd62695253d4c8c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:52.934445 8171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9196497cb11f4ccbbd62695253d4c8c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:52.947568 9138 tablet_bootstrap.cc:492] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:53:52.949631 9138 tablet_bootstrap.cc:654] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:52.958375 9138 tablet_bootstrap.cc:492] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854: No bootstrap required, opened a new log
I20250627 13:53:52.958676 9138 ts_tablet_manager.cc:1397] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20250627 13:53:52.959744 9138 raft_consensus.cc:357] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:52.960145 9138 raft_consensus.cc:383] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:52.960284 9138 raft_consensus.cc:738] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: FOLLOWER
I20250627 13:53:52.960332 9142 tablet_bootstrap.cc:492] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
I20250627 13:53:52.960660 9138 consensus_queue.cc:260] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:52.961679 9138 ts_tablet_manager.cc:1428] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250627 13:53:52.962379 9138 tablet_bootstrap.cc:492] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:53:52.964829 9138 tablet_bootstrap.cc:654] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:52.968566 9142 tablet_bootstrap.cc:654] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:52.969295 9139 tablet_bootstrap.cc:492] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap starting.
I20250627 13:53:52.970451 9138 tablet_bootstrap.cc:492] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854: No bootstrap required, opened a new log
I20250627 13:53:52.970506 9137 tablet_bootstrap.cc:492] T cff879c8e57d42c28440b3bd310f1ba9 P 83574c1bf8ea45adaff939158f93c096: No bootstrap required, opened a new log
I20250627 13:53:52.970690 9137 ts_tablet_manager.cc:1397] T cff879c8e57d42c28440b3bd310f1ba9 P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.040s user 0.024s sys 0.015s
I20250627 13:53:52.970697 9138 ts_tablet_manager.cc:1397] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20250627 13:53:52.971468 9139 tablet_bootstrap.cc:654] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:52.971582 9138 raft_consensus.cc:357] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:52.971596 9137 raft_consensus.cc:357] T cff879c8e57d42c28440b3bd310f1ba9 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:52.971796 9137 raft_consensus.cc:383] T cff879c8e57d42c28440b3bd310f1ba9 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:52.971937 9138 raft_consensus.cc:383] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:52.971932 9137 raft_consensus.cc:738] T cff879c8e57d42c28440b3bd310f1ba9 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: FOLLOWER
I20250627 13:53:52.972079 9138 raft_consensus.cc:738] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: FOLLOWER
I20250627 13:53:52.972213 9137 consensus_queue.cc:260] T cff879c8e57d42c28440b3bd310f1ba9 P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:52.972420 9138 consensus_queue.cc:260] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:52.973037 9138 ts_tablet_manager.cc:1428] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250627 13:53:52.973428 9137 ts_tablet_manager.cc:1428] T cff879c8e57d42c28440b3bd310f1ba9 P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250627 13:53:52.973786 9137 tablet_bootstrap.cc:492] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:53:52.975570 9137 tablet_bootstrap.cc:654] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:52.977936 9142 tablet_bootstrap.cc:492] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e: No bootstrap required, opened a new log
I20250627 13:53:52.978137 9142 ts_tablet_manager.cc:1397] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.007s
I20250627 13:53:52.979025 9142 raft_consensus.cc:357] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:52.979382 9142 raft_consensus.cc:383] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:52.979521 9142 raft_consensus.cc:738] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: FOLLOWER
I20250627 13:53:52.979645 8962 raft_consensus.cc:491] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:52.979818 8962 raft_consensus.cc:513] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:52.979840 9142 consensus_queue.cc:260] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:52.980520 9142 ts_tablet_manager.cc:1428] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250627 13:53:52.980633 8962 leader_election.cc:290] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467), 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:53:52.980806 9142 tablet_bootstrap.cc:492] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
I20250627 13:53:52.982263 7801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cff879c8e57d42c28440b3bd310f1ba9" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096" is_pre_election: true
I20250627 13:53:52.982547 7801 raft_consensus.cc:2466] T cff879c8e57d42c28440b3bd310f1ba9 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a41a89ae8ed743edb2b5cf4490d8c854 in term 0.
I20250627 13:53:52.982748 9142 tablet_bootstrap.cc:654] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:52.983296 8061 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cff879c8e57d42c28440b3bd310f1ba9" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660" is_pre_election: true
I20250627 13:53:52.983891 9139 tablet_bootstrap.cc:492] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660: No bootstrap required, opened a new log
I20250627 13:53:52.984100 9139 ts_tablet_manager.cc:1397] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.004s
W20250627 13:53:52.984503 7866 leader_election.cc:343] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:53:52.984874 7866 leader_election.cc:304] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, a41a89ae8ed743edb2b5cf4490d8c854; no voters: 9cf479bdcd0242738d2a77db4ad9f660
I20250627 13:53:52.985134 9139 raft_consensus.cc:357] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:52.985368 9139 raft_consensus.cc:383] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:52.985474 9139 raft_consensus.cc:738] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Initialized, Role: FOLLOWER
I20250627 13:53:52.985821 8962 raft_consensus.cc:2802] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:52.985755 9139 consensus_queue.cc:260] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:52.985962 8962 raft_consensus.cc:491] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:52.986063 8962 raft_consensus.cc:3058] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:52.986424 9139 ts_tablet_manager.cc:1428] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:53:52.987488 9139 tablet_bootstrap.cc:492] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap starting.
I20250627 13:53:52.991194 9139 tablet_bootstrap.cc:654] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:52.993323 8962 raft_consensus.cc:513] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:52.994048 8962 leader_election.cc:290] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 1 election: Requested vote from peers 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467), 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:53:52.994057 9137 tablet_bootstrap.cc:492] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096: No bootstrap required, opened a new log
I20250627 13:53:52.994252 9137 ts_tablet_manager.cc:1397] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.008s
I20250627 13:53:52.995265 9137 raft_consensus.cc:357] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:52.995504 8061 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cff879c8e57d42c28440b3bd310f1ba9" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660"
I20250627 13:53:52.995747 7801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cff879c8e57d42c28440b3bd310f1ba9" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096"
I20250627 13:53:52.995949 7801 raft_consensus.cc:3058] T cff879c8e57d42c28440b3bd310f1ba9 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:52.996058 9137 raft_consensus.cc:383] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250627 13:53:52.996177 7866 leader_election.cc:343] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:53:52.996196 9137 raft_consensus.cc:738] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: FOLLOWER
I20250627 13:53:52.996563 9137 consensus_queue.cc:260] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:52.998116 9137 ts_tablet_manager.cc:1428] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250627 13:53:52.998515 9137 tablet_bootstrap.cc:492] T 9196497cb11f4ccbbd62695253d4c8c9 P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:53:52.999608 7801 raft_consensus.cc:2466] T cff879c8e57d42c28440b3bd310f1ba9 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41a89ae8ed743edb2b5cf4490d8c854 in term 1.
I20250627 13:53:52.999989 7866 leader_election.cc:304] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, a41a89ae8ed743edb2b5cf4490d8c854; no voters: 9cf479bdcd0242738d2a77db4ad9f660
I20250627 13:53:53.001214 9139 tablet_bootstrap.cc:492] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660: No bootstrap required, opened a new log
I20250627 13:53:53.001416 9139 ts_tablet_manager.cc:1397] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20250627 13:53:53.001479 9137 tablet_bootstrap.cc:654] T 9196497cb11f4ccbbd62695253d4c8c9 P 83574c1bf8ea45adaff939158f93c096: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:53.001754 8962 raft_consensus.cc:2802] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:53.001901 9142 tablet_bootstrap.cc:492] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e: No bootstrap required, opened a new log
I20250627 13:53:53.002132 9142 ts_tablet_manager.cc:1397] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.005s
I20250627 13:53:53.003186 9139 raft_consensus.cc:357] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:53.003209 9142 raft_consensus.cc:357] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:53.003463 9142 raft_consensus.cc:383] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:53.003559 9139 raft_consensus.cc:383] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:53.003645 9142 raft_consensus.cc:738] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: FOLLOWER
I20250627 13:53:53.003719 9139 raft_consensus.cc:738] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Initialized, Role: FOLLOWER
I20250627 13:53:53.004037 9142 consensus_queue.cc:260] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:53.004061 9139 consensus_queue.cc:260] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:53.004803 9139 ts_tablet_manager.cc:1428] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250627 13:53:53.008590 9142 ts_tablet_manager.cc:1428] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250627 13:53:53.010674 8962 raft_consensus.cc:695] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Becoming Leader. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Running, Role: LEADER
I20250627 13:53:53.011075 9137 tablet_bootstrap.cc:492] T 9196497cb11f4ccbbd62695253d4c8c9 P 83574c1bf8ea45adaff939158f93c096: No bootstrap required, opened a new log
I20250627 13:53:53.011083 8962 consensus_queue.cc:237] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:53.011296 9137 ts_tablet_manager.cc:1397] T 9196497cb11f4ccbbd62695253d4c8c9 P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20250627 13:53:53.011893 9137 raft_consensus.cc:357] T 9196497cb11f4ccbbd62695253d4c8c9 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:53.012030 9137 raft_consensus.cc:383] T 9196497cb11f4ccbbd62695253d4c8c9 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:53.012161 9137 raft_consensus.cc:738] T 9196497cb11f4ccbbd62695253d4c8c9 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: FOLLOWER
I20250627 13:53:53.012387 9137 consensus_queue.cc:260] T 9196497cb11f4ccbbd62695253d4c8c9 P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:53.012907 9137 ts_tablet_manager.cc:1428] T 9196497cb11f4ccbbd62695253d4c8c9 P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250627 13:53:53.013772 7661 catalog_manager.cc:5582] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: term changed from 0 to 1, leader changed from <none> to a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:53.015009 8806 raft_consensus.cc:491] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:53.015174 8806 raft_consensus.cc:513] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:53.015713 8806 leader_election.cc:290] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:53:53.016144 7801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9196497cb11f4ccbbd62695253d4c8c9" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096" is_pre_election: true
I20250627 13:53:53.016305 7801 raft_consensus.cc:2466] T 9196497cb11f4ccbbd62695253d4c8c9 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 0.
I20250627 13:53:53.016722 8193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9196497cb11f4ccbbd62695253d4c8c9" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e" is_pre_election: true
I20250627 13:53:53.016820 7996 leader_election.cc:304] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, 9cf479bdcd0242738d2a77db4ad9f660; no voters:
I20250627 13:53:53.016955 8193 raft_consensus.cc:2466] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 0.
I20250627 13:53:53.017256 8806 raft_consensus.cc:2802] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:53.017349 8806 raft_consensus.cc:491] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:53.017445 8806 raft_consensus.cc:3058] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:53.019373 8806 raft_consensus.cc:513] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:53.019897 8806 leader_election.cc:290] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 election: Requested vote from peers e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:53:53.020232 8193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9196497cb11f4ccbbd62695253d4c8c9" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e"
I20250627 13:53:53.020280 7801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9196497cb11f4ccbbd62695253d4c8c9" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096"
I20250627 13:53:53.020408 8193 raft_consensus.cc:3058] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:53.020462 7801 raft_consensus.cc:3058] T 9196497cb11f4ccbbd62695253d4c8c9 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:53.022269 8193 raft_consensus.cc:2466] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 1.
I20250627 13:53:53.022287 7801 raft_consensus.cc:2466] T 9196497cb11f4ccbbd62695253d4c8c9 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 1.
I20250627 13:53:53.022634 7999 leader_election.cc:304] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cf479bdcd0242738d2a77db4ad9f660, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:53.022859 8806 raft_consensus.cc:2802] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:53.023036 8806 raft_consensus.cc:695] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Becoming Leader. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Running, Role: LEADER
I20250627 13:53:53.023281 8806 consensus_queue.cc:237] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:53.025832 7661 catalog_manager.cc:5582] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:53.117025 8947 raft_consensus.cc:491] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:53.117220 8947 raft_consensus.cc:513] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:53.117822 8947 leader_election.cc:290] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:53:53.118296 7932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "897206e017ff4305a8cc9092e30f46b6" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" is_pre_election: true
I20250627 13:53:53.118319 8192 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "897206e017ff4305a8cc9092e30f46b6" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e" is_pre_election: true
I20250627 13:53:53.118517 8192 raft_consensus.cc:2466] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 0.
I20250627 13:53:53.118506 7932 raft_consensus.cc:2466] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 0.
I20250627 13:53:53.118767 8806 raft_consensus.cc:491] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:53.118986 7738 leader_election.cc:304] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, a41a89ae8ed743edb2b5cf4490d8c854, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:53.118988 8806 raft_consensus.cc:513] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:53.119283 8947 raft_consensus.cc:2802] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:53.119381 8947 raft_consensus.cc:491] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:53.119483 8947 raft_consensus.cc:3058] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:53.119899 8806 leader_election.cc:290] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467), a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:53:53.120059 7801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cff879c8e57d42c28440b3bd310f1ba9" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096" is_pre_election: true
I20250627 13:53:53.120371 7801 raft_consensus.cc:2391] T cff879c8e57d42c28440b3bd310f1ba9 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9cf479bdcd0242738d2a77db4ad9f660 in current term 1: Already voted for candidate a41a89ae8ed743edb2b5cf4490d8c854 in this term.
I20250627 13:53:53.120360 7932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cff879c8e57d42c28440b3bd310f1ba9" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" is_pre_election: true
I20250627 13:53:53.120915 7998 leader_election.cc:304] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9cf479bdcd0242738d2a77db4ad9f660; no voters: 83574c1bf8ea45adaff939158f93c096, a41a89ae8ed743edb2b5cf4490d8c854
I20250627 13:53:53.121179 8806 raft_consensus.cc:3058] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:53.122069 8947 raft_consensus.cc:513] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:53.122570 8947 leader_election.cc:290] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 1 election: Requested vote from peers e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:53:53.122915 8192 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "897206e017ff4305a8cc9092e30f46b6" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e"
I20250627 13:53:53.123099 8192 raft_consensus.cc:3058] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:53.123183 7932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "897206e017ff4305a8cc9092e30f46b6" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
I20250627 13:53:53.123355 7932 raft_consensus.cc:3058] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:53.123457 8806 raft_consensus.cc:2747] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250627 13:53:53.125304 7932 raft_consensus.cc:2466] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 1.
I20250627 13:53:53.125361 8192 raft_consensus.cc:2466] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 1.
I20250627 13:53:53.125661 7737 leader_election.cc:304] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, a41a89ae8ed743edb2b5cf4490d8c854; no voters:
I20250627 13:53:53.125871 8952 raft_consensus.cc:2802] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:53.126035 8952 raft_consensus.cc:695] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Becoming Leader. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Running, Role: LEADER
I20250627 13:53:53.126276 8952 consensus_queue.cc:237] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:53.128731 7660 catalog_manager.cc:5582] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83574c1bf8ea45adaff939158f93c096 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:53.435379 9067 consensus_queue.cc:1035] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:53.441906 8806 consensus_queue.cc:1035] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250627 13:53:53.511905 9154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:53:53.512207 9154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:53:53.519825 9154 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20250627 13:53:53.560359 8962 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:53.567215 8962 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:53.676924 8192 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250627 13:53:53.677246 7932 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250627 13:53:53.677901 8952 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:53.678953 8952 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:53.797125 9206 mvcc.cc:204] Tried to move back new op lower bound from 7172228848332509184 to 7172228846086922240. Current Snapshot: MvccSnapshot[applied={T|T < 7172228848332509184}]
INSERT report
rows total: 2000
time total: 282.669 ms
time per row: 0.141334 ms
I20250627 13:53:53.949785 7658 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45288:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250627 13:53:53.958801 8042 tablet_service.cc:1468] Processing CreateTablet for tablet 118d5145b23f43798e1c727f2a9e9f92 (DEFAULT_TABLE table=rebalance_test_table_5 [id=dd79cdfc35164ae6a649d3a77ee397c2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:53.958881 8173 tablet_service.cc:1468] Processing CreateTablet for tablet 118d5145b23f43798e1c727f2a9e9f92 (DEFAULT_TABLE table=rebalance_test_table_5 [id=dd79cdfc35164ae6a649d3a77ee397c2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:53.959275 8173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 118d5145b23f43798e1c727f2a9e9f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:53.959277 8042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 118d5145b23f43798e1c727f2a9e9f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:53.959350 8171 tablet_service.cc:1468] Processing CreateTablet for tablet 8ab02f13e45c48d293e2d2d229d11e27 (DEFAULT_TABLE table=rebalance_test_table_5 [id=dd79cdfc35164ae6a649d3a77ee397c2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:53.959704 8171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ab02f13e45c48d293e2d2d229d11e27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:53.965441 7913 tablet_service.cc:1468] Processing CreateTablet for tablet 8ab02f13e45c48d293e2d2d229d11e27 (DEFAULT_TABLE table=rebalance_test_table_5 [id=dd79cdfc35164ae6a649d3a77ee397c2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:53.965546 7912 tablet_service.cc:1468] Processing CreateTablet for tablet 118d5145b23f43798e1c727f2a9e9f92 (DEFAULT_TABLE table=rebalance_test_table_5 [id=dd79cdfc35164ae6a649d3a77ee397c2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:53.966104 7912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 118d5145b23f43798e1c727f2a9e9f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:53.966368 7913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ab02f13e45c48d293e2d2d229d11e27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:53.968467 9228 tablet_bootstrap.cc:492] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
I20250627 13:53:53.968765 7782 tablet_service.cc:1468] Processing CreateTablet for tablet c322cfebf12c4fef8d88fbd68e7e8b6f (DEFAULT_TABLE table=rebalance_test_table_5 [id=dd79cdfc35164ae6a649d3a77ee397c2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:53.969172 7782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c322cfebf12c4fef8d88fbd68e7e8b6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:53.970201 9228 tablet_bootstrap.cc:654] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:53.972940 8041 tablet_service.cc:1468] Processing CreateTablet for tablet 8ab02f13e45c48d293e2d2d229d11e27 (DEFAULT_TABLE table=rebalance_test_table_5 [id=dd79cdfc35164ae6a649d3a77ee397c2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:53.973057 8043 tablet_service.cc:1468] Processing CreateTablet for tablet c322cfebf12c4fef8d88fbd68e7e8b6f (DEFAULT_TABLE table=rebalance_test_table_5 [id=dd79cdfc35164ae6a649d3a77ee397c2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:53.973340 8041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ab02f13e45c48d293e2d2d229d11e27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:53.973316 7911 tablet_service.cc:1468] Processing CreateTablet for tablet c322cfebf12c4fef8d88fbd68e7e8b6f (DEFAULT_TABLE table=rebalance_test_table_5 [id=dd79cdfc35164ae6a649d3a77ee397c2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:53.973552 8043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c322cfebf12c4fef8d88fbd68e7e8b6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:53.973722 7911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c322cfebf12c4fef8d88fbd68e7e8b6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:53.980238 9231 tablet_bootstrap.cc:492] T c322cfebf12c4fef8d88fbd68e7e8b6f P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap starting.
I20250627 13:53:53.982478 9232 tablet_bootstrap.cc:492] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:53:53.984850 9232 tablet_bootstrap.cc:654] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:53.985936 9228 tablet_bootstrap.cc:492] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e: No bootstrap required, opened a new log
I20250627 13:53:53.986084 9228 ts_tablet_manager.cc:1397] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20250627 13:53:53.986904 9228 raft_consensus.cc:357] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:53.987102 9228 raft_consensus.cc:383] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:53.987215 9228 raft_consensus.cc:738] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: FOLLOWER
I20250627 13:53:53.987221 9230 tablet_bootstrap.cc:492] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:53:53.987443 9228 consensus_queue.cc:260] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:53.988075 9228 ts_tablet_manager.cc:1428] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250627 13:53:53.988338 9231 tablet_bootstrap.cc:654] T c322cfebf12c4fef8d88fbd68e7e8b6f P 9cf479bdcd0242738d2a77db4ad9f660: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:53.988593 9228 tablet_bootstrap.cc:492] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
I20250627 13:53:53.989163 9230 tablet_bootstrap.cc:654] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:53.990294 9228 tablet_bootstrap.cc:654] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:53.993458 9231 tablet_bootstrap.cc:492] T c322cfebf12c4fef8d88fbd68e7e8b6f P 9cf479bdcd0242738d2a77db4ad9f660: No bootstrap required, opened a new log
I20250627 13:53:53.993595 9231 ts_tablet_manager.cc:1397] T c322cfebf12c4fef8d88fbd68e7e8b6f P 9cf479bdcd0242738d2a77db4ad9f660: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20250627 13:53:53.994163 9231 raft_consensus.cc:357] T c322cfebf12c4fef8d88fbd68e7e8b6f P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:53.994315 9231 raft_consensus.cc:383] T c322cfebf12c4fef8d88fbd68e7e8b6f P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:53.994414 9231 raft_consensus.cc:738] T c322cfebf12c4fef8d88fbd68e7e8b6f P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Initialized, Role: FOLLOWER
I20250627 13:53:53.994612 9231 consensus_queue.cc:260] T c322cfebf12c4fef8d88fbd68e7e8b6f P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:53.995134 9231 ts_tablet_manager.cc:1428] T c322cfebf12c4fef8d88fbd68e7e8b6f P 9cf479bdcd0242738d2a77db4ad9f660: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250627 13:53:53.995959 9231 tablet_bootstrap.cc:492] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap starting.
I20250627 13:53:53.996601 9228 tablet_bootstrap.cc:492] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e: No bootstrap required, opened a new log
I20250627 13:53:53.996842 9228 ts_tablet_manager.cc:1397] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20250627 13:53:53.997222 9230 tablet_bootstrap.cc:492] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096: No bootstrap required, opened a new log
I20250627 13:53:53.997345 9232 tablet_bootstrap.cc:492] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854: No bootstrap required, opened a new log
I20250627 13:53:53.997442 9230 ts_tablet_manager.cc:1397] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20250627 13:53:53.997493 9232 ts_tablet_manager.cc:1397] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.004s
I20250627 13:53:53.997628 9228 raft_consensus.cc:357] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:53.998174 9232 raft_consensus.cc:357] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:53.998324 9232 raft_consensus.cc:383] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:53.998409 9232 raft_consensus.cc:738] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: FOLLOWER
I20250627 13:53:53.998627 9232 consensus_queue.cc:260] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:53.999107 9232 ts_tablet_manager.cc:1428] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250627 13:53:53.999343 9231 tablet_bootstrap.cc:654] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:53.999420 9232 tablet_bootstrap.cc:492] T c322cfebf12c4fef8d88fbd68e7e8b6f P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:53:53.999518 9228 raft_consensus.cc:383] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:53.999640 9228 raft_consensus.cc:738] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: FOLLOWER
I20250627 13:53:53.999934 9228 consensus_queue.cc:260] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:54.000434 9228 ts_tablet_manager.cc:1428] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250627 13:53:54.001044 9232 tablet_bootstrap.cc:654] T c322cfebf12c4fef8d88fbd68e7e8b6f P a41a89ae8ed743edb2b5cf4490d8c854: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:54.004738 9231 tablet_bootstrap.cc:492] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660: No bootstrap required, opened a new log
I20250627 13:53:54.004946 9231 ts_tablet_manager.cc:1397] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20250627 13:53:54.005766 9231 raft_consensus.cc:357] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:54.005942 9231 raft_consensus.cc:383] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:54.006024 9231 raft_consensus.cc:738] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Initialized, Role: FOLLOWER
I20250627 13:53:54.006260 9231 consensus_queue.cc:260] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:54.006728 9231 ts_tablet_manager.cc:1428] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250627 13:53:54.007015 9231 tablet_bootstrap.cc:492] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap starting.
I20250627 13:53:54.007295 9230 raft_consensus.cc:357] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:54.007669 9230 raft_consensus.cc:383] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:54.007841 9230 raft_consensus.cc:738] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: FOLLOWER
I20250627 13:53:54.008831 9231 tablet_bootstrap.cc:654] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:54.008209 9230 consensus_queue.cc:260] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:54.010946 9230 ts_tablet_manager.cc:1428] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20250627 13:53:54.011783 9231 tablet_bootstrap.cc:492] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660: No bootstrap required, opened a new log
I20250627 13:53:54.011969 9231 ts_tablet_manager.cc:1397] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20250627 13:53:54.012799 9231 raft_consensus.cc:357] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:54.013037 9231 raft_consensus.cc:383] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:54.013182 9231 raft_consensus.cc:738] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Initialized, Role: FOLLOWER
I20250627 13:53:54.013501 9231 consensus_queue.cc:260] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:54.014146 9231 ts_tablet_manager.cc:1428] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250627 13:53:54.015316 9232 tablet_bootstrap.cc:492] T c322cfebf12c4fef8d88fbd68e7e8b6f P a41a89ae8ed743edb2b5cf4490d8c854: No bootstrap required, opened a new log
I20250627 13:53:54.015496 9232 ts_tablet_manager.cc:1397] T c322cfebf12c4fef8d88fbd68e7e8b6f P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.006s
I20250627 13:53:54.016273 9232 raft_consensus.cc:357] T c322cfebf12c4fef8d88fbd68e7e8b6f P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:54.016471 9232 raft_consensus.cc:383] T c322cfebf12c4fef8d88fbd68e7e8b6f P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:54.016553 9232 raft_consensus.cc:738] T c322cfebf12c4fef8d88fbd68e7e8b6f P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: FOLLOWER
I20250627 13:53:54.016811 9232 consensus_queue.cc:260] T c322cfebf12c4fef8d88fbd68e7e8b6f P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:54.017334 9232 ts_tablet_manager.cc:1428] T c322cfebf12c4fef8d88fbd68e7e8b6f P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250627 13:53:54.017699 9232 tablet_bootstrap.cc:492] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:53:54.019479 9232 tablet_bootstrap.cc:654] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:54.021198 9232 tablet_bootstrap.cc:492] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854: No bootstrap required, opened a new log
I20250627 13:53:54.021301 9232 ts_tablet_manager.cc:1397] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250627 13:53:54.021812 9232 raft_consensus.cc:357] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:54.021940 9232 raft_consensus.cc:383] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:54.022003 9232 raft_consensus.cc:738] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: FOLLOWER
I20250627 13:53:54.022179 9232 consensus_queue.cc:260] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:54.022598 9232 ts_tablet_manager.cc:1428] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250627 13:53:54.071842 9200 raft_consensus.cc:491] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:54.072031 9200 raft_consensus.cc:513] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:54.072731 9200 leader_election.cc:290] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933), a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:53:54.073144 8062 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c322cfebf12c4fef8d88fbd68e7e8b6f" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660" is_pre_election: true
I20250627 13:53:54.073212 7932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c322cfebf12c4fef8d88fbd68e7e8b6f" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" is_pre_election: true
I20250627 13:53:54.073347 8062 raft_consensus.cc:2466] T c322cfebf12c4fef8d88fbd68e7e8b6f P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 0.
I20250627 13:53:54.073426 7932 raft_consensus.cc:2466] T c322cfebf12c4fef8d88fbd68e7e8b6f P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 0.
I20250627 13:53:54.073787 7735 leader_election.cc:304] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, 9cf479bdcd0242738d2a77db4ad9f660; no voters:
I20250627 13:53:54.074168 9200 raft_consensus.cc:2802] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:54.074297 9200 raft_consensus.cc:491] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:54.074438 9200 raft_consensus.cc:3058] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:54.076757 9200 raft_consensus.cc:513] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:54.077210 9200 leader_election.cc:290] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 1 election: Requested vote from peers 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933), a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:53:54.077615 8062 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c322cfebf12c4fef8d88fbd68e7e8b6f" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660"
I20250627 13:53:54.077636 7932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c322cfebf12c4fef8d88fbd68e7e8b6f" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
I20250627 13:53:54.077847 7932 raft_consensus.cc:3058] T c322cfebf12c4fef8d88fbd68e7e8b6f P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:54.077855 8062 raft_consensus.cc:3058] T c322cfebf12c4fef8d88fbd68e7e8b6f P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:54.079711 8062 raft_consensus.cc:2466] T c322cfebf12c4fef8d88fbd68e7e8b6f P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 1.
I20250627 13:53:54.079787 7932 raft_consensus.cc:2466] T c322cfebf12c4fef8d88fbd68e7e8b6f P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 1.
I20250627 13:53:54.080247 7735 leader_election.cc:304] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, 9cf479bdcd0242738d2a77db4ad9f660; no voters:
I20250627 13:53:54.080513 9200 raft_consensus.cc:2802] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:54.080693 9200 raft_consensus.cc:695] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Becoming Leader. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Running, Role: LEADER
I20250627 13:53:54.080988 9200 consensus_queue.cc:237] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:54.083628 7655 catalog_manager.cc:5582] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83574c1bf8ea45adaff939158f93c096 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:54.109544 8806 raft_consensus.cc:491] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:54.109727 8806 raft_consensus.cc:513] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:54.110291 8806 leader_election.cc:290] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:53:54.110662 8193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "118d5145b23f43798e1c727f2a9e9f92" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e" is_pre_election: true
I20250627 13:53:54.110807 7932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "118d5145b23f43798e1c727f2a9e9f92" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" is_pre_election: true
I20250627 13:53:54.110867 8193 raft_consensus.cc:2466] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 0.
I20250627 13:53:54.111044 7932 raft_consensus.cc:2466] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 0.
I20250627 13:53:54.111297 7999 leader_election.cc:304] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cf479bdcd0242738d2a77db4ad9f660, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:54.111565 8806 raft_consensus.cc:2802] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:54.111672 8806 raft_consensus.cc:491] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:54.111737 8806 raft_consensus.cc:3058] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:54.113459 8806 raft_consensus.cc:513] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:54.113950 8806 leader_election.cc:290] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 election: Requested vote from peers e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:53:54.114276 8193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "118d5145b23f43798e1c727f2a9e9f92" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e"
I20250627 13:53:54.114414 8193 raft_consensus.cc:3058] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:54.114385 7932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "118d5145b23f43798e1c727f2a9e9f92" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
I20250627 13:53:54.114579 7932 raft_consensus.cc:3058] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:54.116358 8193 raft_consensus.cc:2466] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 1.
I20250627 13:53:54.116437 7932 raft_consensus.cc:2466] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 1.
I20250627 13:53:54.116750 7999 leader_election.cc:304] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cf479bdcd0242738d2a77db4ad9f660, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:54.117003 8806 raft_consensus.cc:2802] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:54.117143 8806 raft_consensus.cc:695] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Becoming Leader. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Running, Role: LEADER
I20250627 13:53:54.117403 8806 consensus_queue.cc:237] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:54.120039 7655 catalog_manager.cc:5582] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:54.126235 8855 raft_consensus.cc:491] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:54.126428 8855 raft_consensus.cc:513] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:54.126941 8855 leader_election.cc:290] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675), 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:53:54.127350 7932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ab02f13e45c48d293e2d2d229d11e27" candidate_uuid: "e4c831df4da841929d13d6c783099b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" is_pre_election: true
I20250627 13:53:54.127377 8062 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ab02f13e45c48d293e2d2d229d11e27" candidate_uuid: "e4c831df4da841929d13d6c783099b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660" is_pre_election: true
I20250627 13:53:54.127504 7932 raft_consensus.cc:2466] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4c831df4da841929d13d6c783099b0e in term 0.
I20250627 13:53:54.127538 8062 raft_consensus.cc:2466] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4c831df4da841929d13d6c783099b0e in term 0.
I20250627 13:53:54.127934 8126 leader_election.cc:304] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cf479bdcd0242738d2a77db4ad9f660, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:54.128235 8855 raft_consensus.cc:2802] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:54.128336 8855 raft_consensus.cc:491] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:54.128451 8855 raft_consensus.cc:3058] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:54.130937 8855 raft_consensus.cc:513] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:54.131511 8855 leader_election.cc:290] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [CANDIDATE]: Term 1 election: Requested vote from peers a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675), 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:53:54.131783 7932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ab02f13e45c48d293e2d2d229d11e27" candidate_uuid: "e4c831df4da841929d13d6c783099b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
I20250627 13:53:54.131974 7932 raft_consensus.cc:3058] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:54.131968 8062 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ab02f13e45c48d293e2d2d229d11e27" candidate_uuid: "e4c831df4da841929d13d6c783099b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660"
I20250627 13:53:54.132138 8062 raft_consensus.cc:3058] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:54.133965 7932 raft_consensus.cc:2466] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c831df4da841929d13d6c783099b0e in term 1.
I20250627 13:53:54.134280 8128 leader_election.cc:304] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a41a89ae8ed743edb2b5cf4490d8c854, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:54.134518 8855 raft_consensus.cc:2802] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:54.134688 8855 raft_consensus.cc:695] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Becoming Leader. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Running, Role: LEADER
I20250627 13:53:54.134933 8855 consensus_queue.cc:237] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:54.137445 7655 catalog_manager.cc:5582] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e reported cstate change: term changed from 0 to 1, leader changed from <none> to e4c831df4da841929d13d6c783099b0e (127.0.103.68). New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:54.147495 8062 raft_consensus.cc:2466] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c831df4da841929d13d6c783099b0e in term 1.
W20250627 13:53:54.489259 9245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:53:54.489576 9245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:53:54.497742 9245 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20250627 13:53:54.544185 8806 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:54.550225 8806 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:54.560526 8855 consensus_queue.cc:1035] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:54.579767 8855 consensus_queue.cc:1035] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:54.599058 9200 consensus_queue.cc:1035] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:54.614675 9277 consensus_queue.cc:1035] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 272.512 ms
time per row: 0.136256 ms
I20250627 13:53:54.924427 7661 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45288:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250627 13:53:54.932976 8173 tablet_service.cc:1468] Processing CreateTablet for tablet 720cd645f0b7414f819fcbff3fd262c3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f83cab74056a4e7681a701169d1c1716]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:54.933416 8173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 720cd645f0b7414f819fcbff3fd262c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:54.933409 7912 tablet_service.cc:1468] Processing CreateTablet for tablet 720cd645f0b7414f819fcbff3fd262c3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f83cab74056a4e7681a701169d1c1716]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:54.933852 7912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 720cd645f0b7414f819fcbff3fd262c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:54.934247 8172 tablet_service.cc:1468] Processing CreateTablet for tablet 14dba72d45694defb83f41c1572722ca (DEFAULT_TABLE table=rebalance_test_table_6 [id=f83cab74056a4e7681a701169d1c1716]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:54.934619 8172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14dba72d45694defb83f41c1572722ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:54.935868 7781 tablet_service.cc:1468] Processing CreateTablet for tablet 14dba72d45694defb83f41c1572722ca (DEFAULT_TABLE table=rebalance_test_table_6 [id=f83cab74056a4e7681a701169d1c1716]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:54.936261 7781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14dba72d45694defb83f41c1572722ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:54.934511 8171 tablet_service.cc:1468] Processing CreateTablet for tablet e2bd060a4a9e4765b45b0242ae56e489 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f83cab74056a4e7681a701169d1c1716]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:54.937985 7780 tablet_service.cc:1468] Processing CreateTablet for tablet e2bd060a4a9e4765b45b0242ae56e489 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f83cab74056a4e7681a701169d1c1716]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:54.938179 8171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2bd060a4a9e4765b45b0242ae56e489. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:54.938344 7780 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2bd060a4a9e4765b45b0242ae56e489. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:54.940161 7782 tablet_service.cc:1468] Processing CreateTablet for tablet 720cd645f0b7414f819fcbff3fd262c3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f83cab74056a4e7681a701169d1c1716]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:54.940507 7782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 720cd645f0b7414f819fcbff3fd262c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:54.942147 7911 tablet_service.cc:1468] Processing CreateTablet for tablet 14dba72d45694defb83f41c1572722ca (DEFAULT_TABLE table=rebalance_test_table_6 [id=f83cab74056a4e7681a701169d1c1716]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:54.942545 7911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14dba72d45694defb83f41c1572722ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:54.946175 8041 tablet_service.cc:1468] Processing CreateTablet for tablet e2bd060a4a9e4765b45b0242ae56e489 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f83cab74056a4e7681a701169d1c1716]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:54.946657 8041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2bd060a4a9e4765b45b0242ae56e489. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:54.953253 9309 tablet_bootstrap.cc:492] T 720cd645f0b7414f819fcbff3fd262c3 P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:53:54.955811 9309 tablet_bootstrap.cc:654] T 720cd645f0b7414f819fcbff3fd262c3 P a41a89ae8ed743edb2b5cf4490d8c854: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:54.966264 9309 tablet_bootstrap.cc:492] T 720cd645f0b7414f819fcbff3fd262c3 P a41a89ae8ed743edb2b5cf4490d8c854: No bootstrap required, opened a new log
I20250627 13:53:54.966856 9309 ts_tablet_manager.cc:1397] T 720cd645f0b7414f819fcbff3fd262c3 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20250627 13:53:54.967471 9313 tablet_bootstrap.cc:492] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap starting.
I20250627 13:53:54.967741 9311 tablet_bootstrap.cc:492] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:53:54.968847 9309 raft_consensus.cc:357] T 720cd645f0b7414f819fcbff3fd262c3 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:54.969146 9309 raft_consensus.cc:383] T 720cd645f0b7414f819fcbff3fd262c3 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:54.969281 9309 raft_consensus.cc:738] T 720cd645f0b7414f819fcbff3fd262c3 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: FOLLOWER
I20250627 13:53:54.969537 9311 tablet_bootstrap.cc:654] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:54.969662 9309 consensus_queue.cc:260] T 720cd645f0b7414f819fcbff3fd262c3 P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:54.970270 9309 ts_tablet_manager.cc:1428] T 720cd645f0b7414f819fcbff3fd262c3 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250627 13:53:54.970662 9309 tablet_bootstrap.cc:492] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:53:54.970953 9310 tablet_bootstrap.cc:492] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
I20250627 13:53:54.972941 9309 tablet_bootstrap.cc:654] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:54.973344 9310 tablet_bootstrap.cc:654] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:54.975454 9311 tablet_bootstrap.cc:492] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096: No bootstrap required, opened a new log
I20250627 13:53:54.975644 9311 ts_tablet_manager.cc:1397] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20250627 13:53:54.976256 9309 tablet_bootstrap.cc:492] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854: No bootstrap required, opened a new log
I20250627 13:53:54.976420 9313 tablet_bootstrap.cc:654] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:54.976459 9309 ts_tablet_manager.cc:1397] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20250627 13:53:54.976496 9311 raft_consensus.cc:357] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:54.976671 9311 raft_consensus.cc:383] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:54.976768 9311 raft_consensus.cc:738] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: FOLLOWER
I20250627 13:53:54.977058 9311 consensus_queue.cc:260] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:54.977268 9309 raft_consensus.cc:357] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:54.977463 9309 raft_consensus.cc:383] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:54.977550 9309 raft_consensus.cc:738] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: FOLLOWER
I20250627 13:53:54.977794 9309 consensus_queue.cc:260] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:54.978287 9309 ts_tablet_manager.cc:1428] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:53:54.978291 9311 ts_tablet_manager.cc:1428] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250627 13:53:54.978590 9311 tablet_bootstrap.cc:492] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:53:54.980260 9311 tablet_bootstrap.cc:654] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:54.981585 9313 tablet_bootstrap.cc:492] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660: No bootstrap required, opened a new log
I20250627 13:53:54.981902 9310 tablet_bootstrap.cc:492] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e: No bootstrap required, opened a new log
I20250627 13:53:54.982074 9310 ts_tablet_manager.cc:1397] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.004s
I20250627 13:53:54.982214 9311 tablet_bootstrap.cc:492] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096: No bootstrap required, opened a new log
I20250627 13:53:54.982362 9311 ts_tablet_manager.cc:1397] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20250627 13:53:54.982932 9310 raft_consensus.cc:357] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:54.983142 9310 raft_consensus.cc:383] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:54.983073 9311 raft_consensus.cc:357] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:54.983249 9311 raft_consensus.cc:383] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:54.983239 9310 raft_consensus.cc:738] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: FOLLOWER
I20250627 13:53:54.983320 9311 raft_consensus.cc:738] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: FOLLOWER
I20250627 13:53:54.983456 9310 consensus_queue.cc:260] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:54.983542 9311 consensus_queue.cc:260] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:54.983995 9310 ts_tablet_manager.cc:1428] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250627 13:53:54.984194 9313 ts_tablet_manager.cc:1397] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20250627 13:53:54.984627 9311 ts_tablet_manager.cc:1428] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250627 13:53:54.984905 9311 tablet_bootstrap.cc:492] T 14dba72d45694defb83f41c1572722ca P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:53:54.984948 9313 raft_consensus.cc:357] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:54.985113 9313 raft_consensus.cc:383] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:54.985193 9313 raft_consensus.cc:738] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Initialized, Role: FOLLOWER
I20250627 13:53:54.985384 9313 consensus_queue.cc:260] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:54.985985 9313 ts_tablet_manager.cc:1428] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:53:54.986317 9310 tablet_bootstrap.cc:492] T 720cd645f0b7414f819fcbff3fd262c3 P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
I20250627 13:53:54.988040 9311 tablet_bootstrap.cc:654] T 14dba72d45694defb83f41c1572722ca P 83574c1bf8ea45adaff939158f93c096: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:54.988343 9310 tablet_bootstrap.cc:654] T 720cd645f0b7414f819fcbff3fd262c3 P e4c831df4da841929d13d6c783099b0e: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:54.989995 9311 tablet_bootstrap.cc:492] T 14dba72d45694defb83f41c1572722ca P 83574c1bf8ea45adaff939158f93c096: No bootstrap required, opened a new log
I20250627 13:53:54.990140 9311 ts_tablet_manager.cc:1397] T 14dba72d45694defb83f41c1572722ca P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20250627 13:53:54.990716 9311 raft_consensus.cc:357] T 14dba72d45694defb83f41c1572722ca P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:54.990852 9311 raft_consensus.cc:383] T 14dba72d45694defb83f41c1572722ca P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:54.990940 9311 raft_consensus.cc:738] T 14dba72d45694defb83f41c1572722ca P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: FOLLOWER
I20250627 13:53:54.991163 9310 tablet_bootstrap.cc:492] T 720cd645f0b7414f819fcbff3fd262c3 P e4c831df4da841929d13d6c783099b0e: No bootstrap required, opened a new log
I20250627 13:53:54.991153 9311 consensus_queue.cc:260] T 14dba72d45694defb83f41c1572722ca P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:54.991335 9310 ts_tablet_manager.cc:1397] T 720cd645f0b7414f819fcbff3fd262c3 P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20250627 13:53:54.991600 9311 ts_tablet_manager.cc:1428] T 14dba72d45694defb83f41c1572722ca P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250627 13:53:54.992034 9310 raft_consensus.cc:357] T 720cd645f0b7414f819fcbff3fd262c3 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:54.992237 9310 raft_consensus.cc:383] T 720cd645f0b7414f819fcbff3fd262c3 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:54.992370 9310 raft_consensus.cc:738] T 720cd645f0b7414f819fcbff3fd262c3 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: FOLLOWER
I20250627 13:53:54.992679 9310 consensus_queue.cc:260] T 720cd645f0b7414f819fcbff3fd262c3 P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:54.993083 9310 ts_tablet_manager.cc:1428] T 720cd645f0b7414f819fcbff3fd262c3 P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250627 13:53:54.993391 9310 tablet_bootstrap.cc:492] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
I20250627 13:53:54.994910 9310 tablet_bootstrap.cc:654] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:54.997009 9310 tablet_bootstrap.cc:492] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e: No bootstrap required, opened a new log
I20250627 13:53:54.997128 9310 ts_tablet_manager.cc:1397] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250627 13:53:54.997627 9310 raft_consensus.cc:357] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:54.997795 9310 raft_consensus.cc:383] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:54.997885 9310 raft_consensus.cc:738] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: FOLLOWER
I20250627 13:53:54.998106 9310 consensus_queue.cc:260] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:54.998512 9310 ts_tablet_manager.cc:1428] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250627 13:53:55.173087 9296 raft_consensus.cc:491] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:55.173293 9296 raft_consensus.cc:513] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:55.173918 9296 leader_election.cc:290] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675), 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:53:55.174358 7932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14dba72d45694defb83f41c1572722ca" candidate_uuid: "e4c831df4da841929d13d6c783099b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" is_pre_election: true
I20250627 13:53:55.174368 7801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14dba72d45694defb83f41c1572722ca" candidate_uuid: "e4c831df4da841929d13d6c783099b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096" is_pre_election: true
I20250627 13:53:55.174567 7932 raft_consensus.cc:2466] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4c831df4da841929d13d6c783099b0e in term 0.
I20250627 13:53:55.174579 7801 raft_consensus.cc:2466] T 14dba72d45694defb83f41c1572722ca P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4c831df4da841929d13d6c783099b0e in term 0.
I20250627 13:53:55.175017 8126 leader_election.cc:304] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, a41a89ae8ed743edb2b5cf4490d8c854, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:55.175278 9296 raft_consensus.cc:2802] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:55.175388 9296 raft_consensus.cc:491] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:55.175475 9296 raft_consensus.cc:3058] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:55.177619 9296 raft_consensus.cc:513] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:55.178181 9296 leader_election.cc:290] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [CANDIDATE]: Term 1 election: Requested vote from peers a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675), 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:53:55.178479 7932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14dba72d45694defb83f41c1572722ca" candidate_uuid: "e4c831df4da841929d13d6c783099b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
I20250627 13:53:55.178678 7932 raft_consensus.cc:3058] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:55.178845 7801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14dba72d45694defb83f41c1572722ca" candidate_uuid: "e4c831df4da841929d13d6c783099b0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096"
I20250627 13:53:55.179047 7801 raft_consensus.cc:3058] T 14dba72d45694defb83f41c1572722ca P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:55.180945 7932 raft_consensus.cc:2466] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c831df4da841929d13d6c783099b0e in term 1.
I20250627 13:53:55.181327 8128 leader_election.cc:304] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a41a89ae8ed743edb2b5cf4490d8c854, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:55.181612 9296 raft_consensus.cc:2802] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:55.181782 9296 raft_consensus.cc:695] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Becoming Leader. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Running, Role: LEADER
I20250627 13:53:55.182113 9296 consensus_queue.cc:237] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:55.182416 7801 raft_consensus.cc:2466] T 14dba72d45694defb83f41c1572722ca P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c831df4da841929d13d6c783099b0e in term 1.
I20250627 13:53:55.185073 7655 catalog_manager.cc:5582] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e reported cstate change: term changed from 0 to 1, leader changed from <none> to e4c831df4da841929d13d6c783099b0e (127.0.103.68). New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:55.296056 9301 raft_consensus.cc:491] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:55.296226 9301 raft_consensus.cc:513] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:55.296877 9301 leader_election.cc:290] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467), e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:53:55.297201 7801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2bd060a4a9e4765b45b0242ae56e489" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096" is_pre_election: true
I20250627 13:53:55.297420 7801 raft_consensus.cc:2466] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 0.
I20250627 13:53:55.297396 8193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2bd060a4a9e4765b45b0242ae56e489" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e" is_pre_election: true
I20250627 13:53:55.297598 8193 raft_consensus.cc:2466] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 0.
I20250627 13:53:55.297775 7996 leader_election.cc:304] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, 9cf479bdcd0242738d2a77db4ad9f660; no voters:
I20250627 13:53:55.298008 9301 raft_consensus.cc:2802] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:55.298127 9301 raft_consensus.cc:491] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:55.298214 9301 raft_consensus.cc:3058] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:55.299876 9301 raft_consensus.cc:513] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:55.300354 9301 leader_election.cc:290] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 election: Requested vote from peers 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467), e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:53:55.300678 7801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2bd060a4a9e4765b45b0242ae56e489" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096"
I20250627 13:53:55.300729 8193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2bd060a4a9e4765b45b0242ae56e489" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e"
I20250627 13:53:55.300865 7801 raft_consensus.cc:3058] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:55.300880 8193 raft_consensus.cc:3058] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:55.303520 8193 raft_consensus.cc:2466] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 1.
I20250627 13:53:55.303535 7801 raft_consensus.cc:2466] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 1.
I20250627 13:53:55.303913 7996 leader_election.cc:304] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, 9cf479bdcd0242738d2a77db4ad9f660, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:55.304210 9301 raft_consensus.cc:2802] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:55.304334 9301 raft_consensus.cc:695] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Becoming Leader. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Running, Role: LEADER
I20250627 13:53:55.304579 9301 consensus_queue.cc:237] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:55.307325 7655 catalog_manager.cc:5582] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:55.323858 9200 raft_consensus.cc:491] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:55.324026 9200 raft_consensus.cc:513] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:55.324729 9200 leader_election.cc:290] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:53:55.325155 8193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "720cd645f0b7414f819fcbff3fd262c3" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e" is_pre_election: true
I20250627 13:53:55.325377 8193 raft_consensus.cc:2466] T 720cd645f0b7414f819fcbff3fd262c3 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 0.
I20250627 13:53:55.325364 7932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "720cd645f0b7414f819fcbff3fd262c3" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" is_pre_election: true
I20250627 13:53:55.325553 7932 raft_consensus.cc:2466] T 720cd645f0b7414f819fcbff3fd262c3 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 0.
I20250627 13:53:55.325766 7738 leader_election.cc:304] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:55.326053 9200 raft_consensus.cc:2802] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:55.326167 9200 raft_consensus.cc:491] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:55.326247 9200 raft_consensus.cc:3058] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:55.329125 9200 raft_consensus.cc:513] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:55.329929 9200 leader_election.cc:290] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 1 election: Requested vote from peers e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:53:55.330060 7932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "720cd645f0b7414f819fcbff3fd262c3" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
I20250627 13:53:55.330230 7932 raft_consensus.cc:3058] T 720cd645f0b7414f819fcbff3fd262c3 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:55.330232 8192 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "720cd645f0b7414f819fcbff3fd262c3" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e"
I20250627 13:53:55.330377 8192 raft_consensus.cc:3058] T 720cd645f0b7414f819fcbff3fd262c3 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:55.332479 7932 raft_consensus.cc:2466] T 720cd645f0b7414f819fcbff3fd262c3 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 1.
I20250627 13:53:55.332881 8192 raft_consensus.cc:2466] T 720cd645f0b7414f819fcbff3fd262c3 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 1.
I20250627 13:53:55.333261 7738 leader_election.cc:304] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:55.333794 9200 raft_consensus.cc:2802] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:55.333954 9200 raft_consensus.cc:695] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Becoming Leader. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Running, Role: LEADER
I20250627 13:53:55.334199 9200 consensus_queue.cc:237] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:55.336558 7655 catalog_manager.cc:5582] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83574c1bf8ea45adaff939158f93c096 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:55.667114 9296 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:55.674152 9296 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250627 13:53:55.731570 9326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:53:55.731909 9326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:53:55.739370 9326 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20250627 13:53:55.790901 9324 consensus_queue.cc:1035] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:55.796679 9353 consensus_queue.cc:1035] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250627 13:53:55.870599 8806 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:55.876835 8192 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250627 13:53:55.877442 8806 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:55.969604 9328 mvcc.cc:204] Tried to move back new op lower bound from 7172228857343959040 to 7172228855009157120. Current Snapshot: MvccSnapshot[applied={T|T < 7172228857343959040}]
INSERT report
rows total: 2000
time total: 227.208 ms
time per row: 0.113604 ms
I20250627 13:53:56.115082 7658 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45288:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250627 13:53:56.123450 7911 tablet_service.cc:1468] Processing CreateTablet for tablet 91664403eed74badbeba83d69f597e1f (DEFAULT_TABLE table=rebalance_test_table_7 [id=46b91111142d4e388d01cd06d3d41940]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:56.123643 8041 tablet_service.cc:1468] Processing CreateTablet for tablet 91664403eed74badbeba83d69f597e1f (DEFAULT_TABLE table=rebalance_test_table_7 [id=46b91111142d4e388d01cd06d3d41940]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:56.123934 7911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91664403eed74badbeba83d69f597e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:56.124043 8041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91664403eed74badbeba83d69f597e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:56.123989 7780 tablet_service.cc:1468] Processing CreateTablet for tablet 91664403eed74badbeba83d69f597e1f (DEFAULT_TABLE table=rebalance_test_table_7 [id=46b91111142d4e388d01cd06d3d41940]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:56.124398 7780 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91664403eed74badbeba83d69f597e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:56.125444 7912 tablet_service.cc:1468] Processing CreateTablet for tablet 570b134177314b31851951c411a9f4bf (DEFAULT_TABLE table=rebalance_test_table_7 [id=46b91111142d4e388d01cd06d3d41940]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:56.125830 7912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 570b134177314b31851951c411a9f4bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:56.126233 7781 tablet_service.cc:1468] Processing CreateTablet for tablet c2468f4746224f0ab6f4ba4b3a628e35 (DEFAULT_TABLE table=rebalance_test_table_7 [id=46b91111142d4e388d01cd06d3d41940]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:56.126566 7781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2468f4746224f0ab6f4ba4b3a628e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:56.130235 8042 tablet_service.cc:1468] Processing CreateTablet for tablet c2468f4746224f0ab6f4ba4b3a628e35 (DEFAULT_TABLE table=rebalance_test_table_7 [id=46b91111142d4e388d01cd06d3d41940]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:56.130584 8042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2468f4746224f0ab6f4ba4b3a628e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:56.130779 8171 tablet_service.cc:1468] Processing CreateTablet for tablet 570b134177314b31851951c411a9f4bf (DEFAULT_TABLE table=rebalance_test_table_7 [id=46b91111142d4e388d01cd06d3d41940]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:56.131167 8171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 570b134177314b31851951c411a9f4bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:56.132385 8172 tablet_service.cc:1468] Processing CreateTablet for tablet c2468f4746224f0ab6f4ba4b3a628e35 (DEFAULT_TABLE table=rebalance_test_table_7 [id=46b91111142d4e388d01cd06d3d41940]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:56.132720 8172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2468f4746224f0ab6f4ba4b3a628e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:56.138196 8043 tablet_service.cc:1468] Processing CreateTablet for tablet 570b134177314b31851951c411a9f4bf (DEFAULT_TABLE table=rebalance_test_table_7 [id=46b91111142d4e388d01cd06d3d41940]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:56.138664 8043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 570b134177314b31851951c411a9f4bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:56.142814 9392 tablet_bootstrap.cc:492] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap starting.
I20250627 13:53:56.143308 9393 tablet_bootstrap.cc:492] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
I20250627 13:53:56.145206 9393 tablet_bootstrap.cc:654] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:56.147660 9392 tablet_bootstrap.cc:654] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:56.155205 9393 tablet_bootstrap.cc:492] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e: No bootstrap required, opened a new log
I20250627 13:53:56.155443 9393 ts_tablet_manager.cc:1397] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20250627 13:53:56.155789 9394 tablet_bootstrap.cc:492] T 91664403eed74badbeba83d69f597e1f P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:53:56.156564 9393 raft_consensus.cc:357] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:56.156883 9393 raft_consensus.cc:383] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:56.157043 9393 raft_consensus.cc:738] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: FOLLOWER
I20250627 13:53:56.157413 9393 consensus_queue.cc:260] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:56.158531 9394 tablet_bootstrap.cc:654] T 91664403eed74badbeba83d69f597e1f P 83574c1bf8ea45adaff939158f93c096: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:56.159992 9392 tablet_bootstrap.cc:492] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660: No bootstrap required, opened a new log
I20250627 13:53:56.160647 9392 ts_tablet_manager.cc:1397] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.004s
I20250627 13:53:56.161295 9393 ts_tablet_manager.cc:1428] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250627 13:53:56.162338 9392 raft_consensus.cc:357] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:56.163055 9392 raft_consensus.cc:383] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:56.164501 9392 raft_consensus.cc:738] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Initialized, Role: FOLLOWER
I20250627 13:53:56.164948 9392 consensus_queue.cc:260] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:56.166891 9392 ts_tablet_manager.cc:1428] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250627 13:53:56.172277 9393 tablet_bootstrap.cc:492] T c2468f4746224f0ab6f4ba4b3a628e35 P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
I20250627 13:53:56.173108 9392 tablet_bootstrap.cc:492] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap starting.
I20250627 13:53:56.175091 9393 tablet_bootstrap.cc:654] T c2468f4746224f0ab6f4ba4b3a628e35 P e4c831df4da841929d13d6c783099b0e: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:56.175416 9392 tablet_bootstrap.cc:654] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:56.178153 9391 tablet_bootstrap.cc:492] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:53:56.179097 9393 tablet_bootstrap.cc:492] T c2468f4746224f0ab6f4ba4b3a628e35 P e4c831df4da841929d13d6c783099b0e: No bootstrap required, opened a new log
I20250627 13:53:56.179240 9393 ts_tablet_manager.cc:1397] T c2468f4746224f0ab6f4ba4b3a628e35 P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20250627 13:53:56.179790 9391 tablet_bootstrap.cc:654] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:56.179836 9393 raft_consensus.cc:357] T c2468f4746224f0ab6f4ba4b3a628e35 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:56.180119 9393 raft_consensus.cc:383] T c2468f4746224f0ab6f4ba4b3a628e35 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:56.180248 9393 raft_consensus.cc:738] T c2468f4746224f0ab6f4ba4b3a628e35 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: FOLLOWER
I20250627 13:53:56.180593 9393 consensus_queue.cc:260] T c2468f4746224f0ab6f4ba4b3a628e35 P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:56.181926 9393 ts_tablet_manager.cc:1428] T c2468f4746224f0ab6f4ba4b3a628e35 P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250627 13:53:56.182341 9391 tablet_bootstrap.cc:492] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854: No bootstrap required, opened a new log
I20250627 13:53:56.182466 9391 ts_tablet_manager.cc:1397] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250627 13:53:56.183145 9391 raft_consensus.cc:357] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:56.183319 9391 raft_consensus.cc:383] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:56.183467 9391 raft_consensus.cc:738] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: FOLLOWER
I20250627 13:53:56.183727 9391 consensus_queue.cc:260] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:56.184844 9391 ts_tablet_manager.cc:1428] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:53:56.185076 9391 tablet_bootstrap.cc:492] T 570b134177314b31851951c411a9f4bf P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:53:56.187291 9391 tablet_bootstrap.cc:654] T 570b134177314b31851951c411a9f4bf P a41a89ae8ed743edb2b5cf4490d8c854: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:56.188980 9391 tablet_bootstrap.cc:492] T 570b134177314b31851951c411a9f4bf P a41a89ae8ed743edb2b5cf4490d8c854: No bootstrap required, opened a new log
I20250627 13:53:56.189123 9391 ts_tablet_manager.cc:1397] T 570b134177314b31851951c411a9f4bf P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250627 13:53:56.189842 9391 raft_consensus.cc:357] T 570b134177314b31851951c411a9f4bf P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:56.190094 9391 raft_consensus.cc:383] T 570b134177314b31851951c411a9f4bf P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:56.190184 9391 raft_consensus.cc:738] T 570b134177314b31851951c411a9f4bf P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: FOLLOWER
I20250627 13:53:56.190323 9394 tablet_bootstrap.cc:492] T 91664403eed74badbeba83d69f597e1f P 83574c1bf8ea45adaff939158f93c096: No bootstrap required, opened a new log
I20250627 13:53:56.190481 9394 ts_tablet_manager.cc:1397] T 91664403eed74badbeba83d69f597e1f P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.035s user 0.007s sys 0.011s
I20250627 13:53:56.190508 9392 tablet_bootstrap.cc:492] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660: No bootstrap required, opened a new log
I20250627 13:53:56.190426 9391 consensus_queue.cc:260] T 570b134177314b31851951c411a9f4bf P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:56.190658 9392 ts_tablet_manager.cc:1397] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20250627 13:53:56.191239 9394 raft_consensus.cc:357] T 91664403eed74badbeba83d69f597e1f P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:56.191270 9392 raft_consensus.cc:357] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:56.191466 9392 raft_consensus.cc:383] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:56.191502 9391 ts_tablet_manager.cc:1428] T 570b134177314b31851951c411a9f4bf P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:53:56.191502 9394 raft_consensus.cc:383] T 91664403eed74badbeba83d69f597e1f P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:56.191591 9392 raft_consensus.cc:738] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Initialized, Role: FOLLOWER
I20250627 13:53:56.191668 9394 raft_consensus.cc:738] T 91664403eed74badbeba83d69f597e1f P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: FOLLOWER
I20250627 13:53:56.191825 9392 consensus_queue.cc:260] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:56.191938 9394 consensus_queue.cc:260] T 91664403eed74badbeba83d69f597e1f P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:56.192358 9392 ts_tablet_manager.cc:1428] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250627 13:53:56.192473 9394 ts_tablet_manager.cc:1428] T 91664403eed74badbeba83d69f597e1f P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250627 13:53:56.192739 9394 tablet_bootstrap.cc:492] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:53:56.192795 9392 tablet_bootstrap.cc:492] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap starting.
I20250627 13:53:56.194443 9394 tablet_bootstrap.cc:654] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:56.195806 9392 tablet_bootstrap.cc:654] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:56.199877 9392 tablet_bootstrap.cc:492] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660: No bootstrap required, opened a new log
I20250627 13:53:56.200022 9392 ts_tablet_manager.cc:1397] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20250627 13:53:56.200668 9392 raft_consensus.cc:357] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:56.200837 9392 raft_consensus.cc:383] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:56.200927 9392 raft_consensus.cc:738] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Initialized, Role: FOLLOWER
I20250627 13:53:56.201148 9392 consensus_queue.cc:260] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:56.201707 9392 ts_tablet_manager.cc:1428] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250627 13:53:56.202718 9394 tablet_bootstrap.cc:492] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096: No bootstrap required, opened a new log
I20250627 13:53:56.202898 9394 ts_tablet_manager.cc:1397] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20250627 13:53:56.203680 9394 raft_consensus.cc:357] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:56.203857 9394 raft_consensus.cc:383] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:56.203931 9394 raft_consensus.cc:738] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: FOLLOWER
I20250627 13:53:56.204164 9394 consensus_queue.cc:260] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:56.204685 9394 ts_tablet_manager.cc:1428] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:53:56.252595 9301 raft_consensus.cc:491] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:56.252784 9301 raft_consensus.cc:513] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:56.253343 9301 leader_election.cc:290] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:53:56.253803 8192 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "570b134177314b31851951c411a9f4bf" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e" is_pre_election: true
I20250627 13:53:56.253916 7933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "570b134177314b31851951c411a9f4bf" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" is_pre_election: true
I20250627 13:53:56.254026 8192 raft_consensus.cc:2466] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 0.
I20250627 13:53:56.254122 7933 raft_consensus.cc:2466] T 570b134177314b31851951c411a9f4bf P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 0.
I20250627 13:53:56.254606 7998 leader_election.cc:304] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9cf479bdcd0242738d2a77db4ad9f660, a41a89ae8ed743edb2b5cf4490d8c854, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:56.254899 9301 raft_consensus.cc:2802] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:56.255024 9301 raft_consensus.cc:491] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:56.255132 9301 raft_consensus.cc:3058] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:56.257004 9301 raft_consensus.cc:513] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:56.257524 9301 leader_election.cc:290] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 election: Requested vote from peers e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:53:56.257932 8192 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "570b134177314b31851951c411a9f4bf" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e"
I20250627 13:53:56.257953 7933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "570b134177314b31851951c411a9f4bf" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
I20250627 13:53:56.258096 8192 raft_consensus.cc:3058] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:56.258131 7933 raft_consensus.cc:3058] T 570b134177314b31851951c411a9f4bf P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:56.259835 8192 raft_consensus.cc:2466] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 1.
I20250627 13:53:56.259864 7933 raft_consensus.cc:2466] T 570b134177314b31851951c411a9f4bf P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 1.
I20250627 13:53:56.260221 7998 leader_election.cc:304] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cf479bdcd0242738d2a77db4ad9f660, a41a89ae8ed743edb2b5cf4490d8c854; no voters:
I20250627 13:53:56.260437 9301 raft_consensus.cc:2802] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:56.260573 9301 raft_consensus.cc:695] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Becoming Leader. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Running, Role: LEADER
I20250627 13:53:56.260794 9301 consensus_queue.cc:237] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:56.263450 7658 catalog_manager.cc:5582] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:56.347606 8806 raft_consensus.cc:491] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:56.347788 8806 raft_consensus.cc:513] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:56.348423 8806 leader_election.cc:290] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:53:56.349072 8193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2468f4746224f0ab6f4ba4b3a628e35" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e" is_pre_election: true
I20250627 13:53:56.349304 8193 raft_consensus.cc:2466] T c2468f4746224f0ab6f4ba4b3a628e35 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 0.
I20250627 13:53:56.349792 7802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2468f4746224f0ab6f4ba4b3a628e35" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096" is_pre_election: true
I20250627 13:53:56.350003 7802 raft_consensus.cc:2466] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 0.
I20250627 13:53:56.350637 7996 leader_election.cc:304] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, 9cf479bdcd0242738d2a77db4ad9f660; no voters:
I20250627 13:53:56.350991 8806 raft_consensus.cc:2802] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:56.351104 8806 raft_consensus.cc:491] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:56.351184 8806 raft_consensus.cc:3058] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:56.354166 8806 raft_consensus.cc:513] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:56.355194 8193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2468f4746224f0ab6f4ba4b3a628e35" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e"
I20250627 13:53:56.355228 7802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2468f4746224f0ab6f4ba4b3a628e35" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096"
I20250627 13:53:56.355384 8193 raft_consensus.cc:3058] T c2468f4746224f0ab6f4ba4b3a628e35 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:56.355403 7802 raft_consensus.cc:3058] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:56.357481 8806 leader_election.cc:290] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 election: Requested vote from peers e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:53:56.358171 7802 raft_consensus.cc:2466] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 1.
I20250627 13:53:56.358268 8193 raft_consensus.cc:2466] T c2468f4746224f0ab6f4ba4b3a628e35 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 1.
I20250627 13:53:56.358552 7996 leader_election.cc:304] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, 9cf479bdcd0242738d2a77db4ad9f660; no voters:
I20250627 13:53:56.358870 8806 raft_consensus.cc:2802] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:56.359019 8806 raft_consensus.cc:695] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Becoming Leader. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Running, Role: LEADER
I20250627 13:53:56.359259 8806 consensus_queue.cc:237] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:56.362048 7658 catalog_manager.cc:5582] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:56.376904 8806 raft_consensus.cc:491] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:56.377094 8806 raft_consensus.cc:513] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:56.377631 8806 leader_election.cc:290] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675), 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:53:56.378047 7933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91664403eed74badbeba83d69f597e1f" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" is_pre_election: true
I20250627 13:53:56.378155 7802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91664403eed74badbeba83d69f597e1f" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096" is_pre_election: true
I20250627 13:53:56.378348 7802 raft_consensus.cc:2466] T 91664403eed74badbeba83d69f597e1f P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 0.
I20250627 13:53:56.378347 7933 raft_consensus.cc:2466] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 0.
I20250627 13:53:56.378697 7998 leader_election.cc:304] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cf479bdcd0242738d2a77db4ad9f660, a41a89ae8ed743edb2b5cf4490d8c854; no voters:
I20250627 13:53:56.378962 8806 raft_consensus.cc:2802] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:56.379060 8806 raft_consensus.cc:491] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:56.379168 8806 raft_consensus.cc:3058] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:56.381004 8806 raft_consensus.cc:513] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:56.381500 8806 leader_election.cc:290] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 election: Requested vote from peers a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675), 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:53:56.381856 7933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91664403eed74badbeba83d69f597e1f" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
I20250627 13:53:56.381938 7802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91664403eed74badbeba83d69f597e1f" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096"
I20250627 13:53:56.382025 7933 raft_consensus.cc:3058] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:56.382120 7802 raft_consensus.cc:3058] T 91664403eed74badbeba83d69f597e1f P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:56.384930 7933 raft_consensus.cc:2466] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 1.
I20250627 13:53:56.384959 7802 raft_consensus.cc:2466] T 91664403eed74badbeba83d69f597e1f P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 1.
I20250627 13:53:56.385329 7996 leader_election.cc:304] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, 9cf479bdcd0242738d2a77db4ad9f660, a41a89ae8ed743edb2b5cf4490d8c854; no voters:
I20250627 13:53:56.385571 8806 raft_consensus.cc:2802] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:56.385792 8806 raft_consensus.cc:695] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Becoming Leader. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Running, Role: LEADER
I20250627 13:53:56.386039 8806 consensus_queue.cc:237] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } }
I20250627 13:53:56.388522 7658 catalog_manager.cc:5582] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:56.641736 8806 consensus_queue.cc:1035] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:56.652166 8806 consensus_queue.cc:1035] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250627 13:53:56.781311 9407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:53:56.781718 9407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:53:56.791828 9407 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20250627 13:53:56.899649 7802 raft_consensus.cc:1273] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250627 13:53:56.900296 8806 consensus_queue.cc:1035] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:56.901696 8193 raft_consensus.cc:1273] T c2468f4746224f0ab6f4ba4b3a628e35 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250627 13:53:56.906284 8806 consensus_queue.cc:1035] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:56.935916 8806 consensus_queue.cc:1035] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250627 13:53:57.017827 7998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.66:43675, user_credentials={real_user=slave}} blocked reactor thread for 90700.9us
I20250627 13:53:57.019469 7933 raft_consensus.cc:1273] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250627 13:53:57.026613 9369 consensus_queue.cc:1035] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 213.378 ms
time per row: 0.106689 ms
-----------------------------------------------------
Suppressions used:
count bytes template
6 1788 libcrypto.so.1.1
-----------------------------------------------------
I20250627 13:53:58.179287 7656 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45288:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250627 13:53:58.188823 7911 tablet_service.cc:1468] Processing CreateTablet for tablet 72db4bf98ab04b7189640f658d8fdcd7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=b84f3fe3fd7c4f01902547d2db982f90]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:58.189236 7911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72db4bf98ab04b7189640f658d8fdcd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:58.189343 8172 tablet_service.cc:1468] Processing CreateTablet for tablet ff83fdbab2b9417eac50495c54f0da54 (DEFAULT_TABLE table=rebalance_test_table_8 [id=b84f3fe3fd7c4f01902547d2db982f90]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:58.189736 8172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff83fdbab2b9417eac50495c54f0da54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:58.193849 9472 tablet_bootstrap.cc:492] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:53:58.194559 8043 tablet_service.cc:1468] Processing CreateTablet for tablet 72db4bf98ab04b7189640f658d8fdcd7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=b84f3fe3fd7c4f01902547d2db982f90]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:58.194628 7781 tablet_service.cc:1468] Processing CreateTablet for tablet 72db4bf98ab04b7189640f658d8fdcd7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=b84f3fe3fd7c4f01902547d2db982f90]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:58.194965 8043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72db4bf98ab04b7189640f658d8fdcd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:58.195063 7781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72db4bf98ab04b7189640f658d8fdcd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:58.195688 9472 tablet_bootstrap.cc:654] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:58.197494 9472 tablet_bootstrap.cc:492] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854: No bootstrap required, opened a new log
I20250627 13:53:58.197624 9472 ts_tablet_manager.cc:1397] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250627 13:53:58.198402 9472 raft_consensus.cc:357] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:58.198570 9472 raft_consensus.cc:383] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:58.198649 9472 raft_consensus.cc:738] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: FOLLOWER
I20250627 13:53:58.198875 9472 consensus_queue.cc:260] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:58.200114 9472 ts_tablet_manager.cc:1428] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:53:58.200280 9474 tablet_bootstrap.cc:492] T 72db4bf98ab04b7189640f658d8fdcd7 P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap starting.
I20250627 13:53:58.194625 7780 tablet_service.cc:1468] Processing CreateTablet for tablet ff83fdbab2b9417eac50495c54f0da54 (DEFAULT_TABLE table=rebalance_test_table_8 [id=b84f3fe3fd7c4f01902547d2db982f90]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:58.200896 9475 tablet_bootstrap.cc:492] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:53:58.200942 7780 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff83fdbab2b9417eac50495c54f0da54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:58.201836 9474 tablet_bootstrap.cc:654] T 72db4bf98ab04b7189640f658d8fdcd7 P 9cf479bdcd0242738d2a77db4ad9f660: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:58.203444 8171 tablet_service.cc:1468] Processing CreateTablet for tablet 00feb28e9047435dbe36a595b1e23fdd (DEFAULT_TABLE table=rebalance_test_table_8 [id=b84f3fe3fd7c4f01902547d2db982f90]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:58.203820 8171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00feb28e9047435dbe36a595b1e23fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:58.206158 7782 tablet_service.cc:1468] Processing CreateTablet for tablet 00feb28e9047435dbe36a595b1e23fdd (DEFAULT_TABLE table=rebalance_test_table_8 [id=b84f3fe3fd7c4f01902547d2db982f90]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:58.206667 7782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00feb28e9047435dbe36a595b1e23fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:58.207486 7911 tablet_service.cc:1468] Processing CreateTablet for tablet ff83fdbab2b9417eac50495c54f0da54 (DEFAULT_TABLE table=rebalance_test_table_8 [id=b84f3fe3fd7c4f01902547d2db982f90]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:58.207918 7911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff83fdbab2b9417eac50495c54f0da54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:58.210635 9475 tablet_bootstrap.cc:654] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:58.212554 8041 tablet_service.cc:1468] Processing CreateTablet for tablet 00feb28e9047435dbe36a595b1e23fdd (DEFAULT_TABLE table=rebalance_test_table_8 [id=b84f3fe3fd7c4f01902547d2db982f90]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:58.213042 8041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00feb28e9047435dbe36a595b1e23fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:58.221762 9472 tablet_bootstrap.cc:492] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:53:58.223399 9472 tablet_bootstrap.cc:654] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:58.228281 9474 tablet_bootstrap.cc:492] T 72db4bf98ab04b7189640f658d8fdcd7 P 9cf479bdcd0242738d2a77db4ad9f660: No bootstrap required, opened a new log
I20250627 13:53:58.228483 9474 ts_tablet_manager.cc:1397] T 72db4bf98ab04b7189640f658d8fdcd7 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.026s
I20250627 13:53:58.228837 9475 tablet_bootstrap.cc:492] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096: No bootstrap required, opened a new log
I20250627 13:53:58.229441 9475 ts_tablet_manager.cc:1397] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.000s
I20250627 13:53:58.229378 9474 raft_consensus.cc:357] T 72db4bf98ab04b7189640f658d8fdcd7 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:58.229673 9474 raft_consensus.cc:383] T 72db4bf98ab04b7189640f658d8fdcd7 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:58.229840 9474 raft_consensus.cc:738] T 72db4bf98ab04b7189640f658d8fdcd7 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Initialized, Role: FOLLOWER
I20250627 13:53:58.229975 9472 tablet_bootstrap.cc:492] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854: No bootstrap required, opened a new log
I20250627 13:53:58.230177 9472 ts_tablet_manager.cc:1397] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20250627 13:53:58.230168 9474 consensus_queue.cc:260] T 72db4bf98ab04b7189640f658d8fdcd7 P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:58.230551 9475 raft_consensus.cc:357] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:58.230844 9475 raft_consensus.cc:383] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:58.230914 9472 raft_consensus.cc:357] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:58.231009 9475 raft_consensus.cc:738] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: FOLLOWER
I20250627 13:53:58.231189 9472 raft_consensus.cc:383] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:58.231319 9472 raft_consensus.cc:738] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: FOLLOWER
I20250627 13:53:58.231292 9475 consensus_queue.cc:260] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:58.231623 9472 consensus_queue.cc:260] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:58.232230 9472 ts_tablet_manager.cc:1428] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:53:58.233700 9474 ts_tablet_manager.cc:1428] T 72db4bf98ab04b7189640f658d8fdcd7 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250627 13:53:58.234189 9479 tablet_bootstrap.cc:492] T ff83fdbab2b9417eac50495c54f0da54 P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
I20250627 13:53:58.235157 9474 tablet_bootstrap.cc:492] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap starting.
I20250627 13:53:58.236030 9479 tablet_bootstrap.cc:654] T ff83fdbab2b9417eac50495c54f0da54 P e4c831df4da841929d13d6c783099b0e: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:58.236076 9475 ts_tablet_manager.cc:1428] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250627 13:53:58.237119 9474 tablet_bootstrap.cc:654] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:58.239253 9475 tablet_bootstrap.cc:492] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:53:58.239789 9479 tablet_bootstrap.cc:492] T ff83fdbab2b9417eac50495c54f0da54 P e4c831df4da841929d13d6c783099b0e: No bootstrap required, opened a new log
I20250627 13:53:58.239952 9479 ts_tablet_manager.cc:1397] T ff83fdbab2b9417eac50495c54f0da54 P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20250627 13:53:58.240435 9475 tablet_bootstrap.cc:654] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:58.240859 9479 raft_consensus.cc:357] T ff83fdbab2b9417eac50495c54f0da54 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:58.241161 9479 raft_consensus.cc:383] T ff83fdbab2b9417eac50495c54f0da54 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:58.241302 9479 raft_consensus.cc:738] T ff83fdbab2b9417eac50495c54f0da54 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: FOLLOWER
I20250627 13:53:58.241544 9479 consensus_queue.cc:260] T ff83fdbab2b9417eac50495c54f0da54 P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:58.242055 9475 tablet_bootstrap.cc:492] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096: No bootstrap required, opened a new log
I20250627 13:53:58.242071 9474 tablet_bootstrap.cc:492] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660: No bootstrap required, opened a new log
I20250627 13:53:58.242226 9474 ts_tablet_manager.cc:1397] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20250627 13:53:58.242228 9475 ts_tablet_manager.cc:1397] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250627 13:53:58.242815 9474 raft_consensus.cc:357] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:58.243033 9474 raft_consensus.cc:383] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:58.243085 9479 ts_tablet_manager.cc:1428] T ff83fdbab2b9417eac50495c54f0da54 P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250627 13:53:58.243141 9474 raft_consensus.cc:738] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Initialized, Role: FOLLOWER
I20250627 13:53:58.243420 9479 tablet_bootstrap.cc:492] T 00feb28e9047435dbe36a595b1e23fdd P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
I20250627 13:53:58.243412 9474 consensus_queue.cc:260] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:58.243908 9474 ts_tablet_manager.cc:1428] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:53:58.244552 9475 raft_consensus.cc:357] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:58.244752 9475 raft_consensus.cc:383] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:58.244822 9475 raft_consensus.cc:738] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: FOLLOWER
I20250627 13:53:58.245153 9479 tablet_bootstrap.cc:654] T 00feb28e9047435dbe36a595b1e23fdd P e4c831df4da841929d13d6c783099b0e: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:58.245216 9475 consensus_queue.cc:260] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:58.246069 9475 ts_tablet_manager.cc:1428] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250627 13:53:58.246390 9475 tablet_bootstrap.cc:492] T 00feb28e9047435dbe36a595b1e23fdd P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:53:58.247957 9475 tablet_bootstrap.cc:654] T 00feb28e9047435dbe36a595b1e23fdd P 83574c1bf8ea45adaff939158f93c096: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:58.249055 9479 tablet_bootstrap.cc:492] T 00feb28e9047435dbe36a595b1e23fdd P e4c831df4da841929d13d6c783099b0e: No bootstrap required, opened a new log
I20250627 13:53:58.249259 9479 ts_tablet_manager.cc:1397] T 00feb28e9047435dbe36a595b1e23fdd P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.005s
I20250627 13:53:58.250094 9479 raft_consensus.cc:357] T 00feb28e9047435dbe36a595b1e23fdd P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:58.250366 9479 raft_consensus.cc:383] T 00feb28e9047435dbe36a595b1e23fdd P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:58.250483 9479 raft_consensus.cc:738] T 00feb28e9047435dbe36a595b1e23fdd P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: FOLLOWER
I20250627 13:53:58.250787 9479 consensus_queue.cc:260] T 00feb28e9047435dbe36a595b1e23fdd P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:58.251379 9479 ts_tablet_manager.cc:1428] T 00feb28e9047435dbe36a595b1e23fdd P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250627 13:53:58.252544 9475 tablet_bootstrap.cc:492] T 00feb28e9047435dbe36a595b1e23fdd P 83574c1bf8ea45adaff939158f93c096: No bootstrap required, opened a new log
I20250627 13:53:58.252697 9475 ts_tablet_manager.cc:1397] T 00feb28e9047435dbe36a595b1e23fdd P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20250627 13:53:58.253422 9475 raft_consensus.cc:357] T 00feb28e9047435dbe36a595b1e23fdd P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:58.253631 9475 raft_consensus.cc:383] T 00feb28e9047435dbe36a595b1e23fdd P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:58.253762 9475 raft_consensus.cc:738] T 00feb28e9047435dbe36a595b1e23fdd P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: FOLLOWER
I20250627 13:53:58.253976 9475 consensus_queue.cc:260] T 00feb28e9047435dbe36a595b1e23fdd P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:58.254431 9475 ts_tablet_manager.cc:1428] T 00feb28e9047435dbe36a595b1e23fdd P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250627 13:53:58.387425 9180 raft_consensus.cc:491] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:58.387635 9180 raft_consensus.cc:513] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:58.388290 9180 leader_election.cc:290] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467), 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:53:58.388597 7802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72db4bf98ab04b7189640f658d8fdcd7" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096" is_pre_election: true
I20250627 13:53:58.388749 8062 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72db4bf98ab04b7189640f658d8fdcd7" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660" is_pre_election: true
I20250627 13:53:58.388837 7802 raft_consensus.cc:2466] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a41a89ae8ed743edb2b5cf4490d8c854 in term 0.
I20250627 13:53:58.388972 8062 raft_consensus.cc:2466] T 72db4bf98ab04b7189640f658d8fdcd7 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a41a89ae8ed743edb2b5cf4490d8c854 in term 0.
I20250627 13:53:58.389253 7866 leader_election.cc:304] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, a41a89ae8ed743edb2b5cf4490d8c854; no voters:
I20250627 13:53:58.389513 9180 raft_consensus.cc:2802] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:58.389631 9180 raft_consensus.cc:491] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:58.389747 9180 raft_consensus.cc:3058] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:58.391631 9180 raft_consensus.cc:513] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:58.392094 9180 leader_election.cc:290] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 1 election: Requested vote from peers 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467), 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:53:58.392486 7802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72db4bf98ab04b7189640f658d8fdcd7" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096"
I20250627 13:53:58.392557 8062 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "72db4bf98ab04b7189640f658d8fdcd7" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660"
I20250627 13:53:58.392694 7802 raft_consensus.cc:3058] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:58.392724 8062 raft_consensus.cc:3058] T 72db4bf98ab04b7189640f658d8fdcd7 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:58.395510 8062 raft_consensus.cc:2466] T 72db4bf98ab04b7189640f658d8fdcd7 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41a89ae8ed743edb2b5cf4490d8c854 in term 1.
I20250627 13:53:58.395510 7802 raft_consensus.cc:2466] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41a89ae8ed743edb2b5cf4490d8c854 in term 1.
I20250627 13:53:58.395881 7866 leader_election.cc:304] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, a41a89ae8ed743edb2b5cf4490d8c854; no voters:
I20250627 13:53:58.396178 9180 raft_consensus.cc:2802] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:58.396513 9180 raft_consensus.cc:695] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Becoming Leader. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Running, Role: LEADER
I20250627 13:53:58.396799 9180 consensus_queue.cc:237] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:58.399977 7655 catalog_manager.cc:5582] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: term changed from 0 to 1, leader changed from <none> to a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:58.577706 9444 raft_consensus.cc:491] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:58.577888 9444 raft_consensus.cc:513] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:58.578428 9444 leader_election.cc:290] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:53:58.578739 8193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00feb28e9047435dbe36a595b1e23fdd" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e" is_pre_election: true
I20250627 13:53:58.578828 7802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00feb28e9047435dbe36a595b1e23fdd" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096" is_pre_election: true
I20250627 13:53:58.578919 8193 raft_consensus.cc:2466] T 00feb28e9047435dbe36a595b1e23fdd P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 0.
I20250627 13:53:58.579023 7802 raft_consensus.cc:2466] T 00feb28e9047435dbe36a595b1e23fdd P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 0.
I20250627 13:53:58.579334 7999 leader_election.cc:304] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, 9cf479bdcd0242738d2a77db4ad9f660, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:58.579577 9444 raft_consensus.cc:2802] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:58.579679 9444 raft_consensus.cc:491] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:58.579777 9444 raft_consensus.cc:3058] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:58.581632 9444 raft_consensus.cc:513] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:58.582180 9444 leader_election.cc:290] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 election: Requested vote from peers e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:53:58.582521 8193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00feb28e9047435dbe36a595b1e23fdd" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e"
I20250627 13:53:58.582713 8193 raft_consensus.cc:3058] T 00feb28e9047435dbe36a595b1e23fdd P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:58.582682 7802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00feb28e9047435dbe36a595b1e23fdd" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096"
I20250627 13:53:58.582826 7802 raft_consensus.cc:3058] T 00feb28e9047435dbe36a595b1e23fdd P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:58.585492 8193 raft_consensus.cc:2466] T 00feb28e9047435dbe36a595b1e23fdd P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 1.
I20250627 13:53:58.585507 7802 raft_consensus.cc:2466] T 00feb28e9047435dbe36a595b1e23fdd P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 1.
I20250627 13:53:58.585937 7996 leader_election.cc:304] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, 9cf479bdcd0242738d2a77db4ad9f660, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:58.586195 9444 raft_consensus.cc:2802] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:58.586310 9444 raft_consensus.cc:695] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Becoming Leader. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Running, Role: LEADER
I20250627 13:53:58.586575 9444 consensus_queue.cc:237] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:58.589318 7655 catalog_manager.cc:5582] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } }
I20250627 13:53:58.675235 9454 raft_consensus.cc:491] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:58.675423 9454 raft_consensus.cc:513] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:58.676165 9454 leader_election.cc:290] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:53:58.676560 8193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff83fdbab2b9417eac50495c54f0da54" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e" is_pre_election: true
I20250627 13:53:58.676692 7933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff83fdbab2b9417eac50495c54f0da54" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" is_pre_election: true
I20250627 13:53:58.676803 8193 raft_consensus.cc:2466] T ff83fdbab2b9417eac50495c54f0da54 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 0.
I20250627 13:53:58.676910 7933 raft_consensus.cc:2466] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 0.
I20250627 13:53:58.677179 7738 leader_election.cc:304] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:58.677446 9454 raft_consensus.cc:2802] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:58.677570 9454 raft_consensus.cc:491] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:58.677701 9454 raft_consensus.cc:3058] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:58.679531 9454 raft_consensus.cc:513] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:58.679952 9454 leader_election.cc:290] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 1 election: Requested vote from peers e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:53:58.680300 8193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff83fdbab2b9417eac50495c54f0da54" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e"
I20250627 13:53:58.680331 7933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff83fdbab2b9417eac50495c54f0da54" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
I20250627 13:53:58.680469 8193 raft_consensus.cc:3058] T ff83fdbab2b9417eac50495c54f0da54 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:58.680513 7933 raft_consensus.cc:3058] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:58.682495 7933 raft_consensus.cc:2466] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 1.
I20250627 13:53:58.682619 8193 raft_consensus.cc:2466] T ff83fdbab2b9417eac50495c54f0da54 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 1.
I20250627 13:53:58.682864 7737 leader_election.cc:304] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, a41a89ae8ed743edb2b5cf4490d8c854; no voters:
I20250627 13:53:58.683120 9454 raft_consensus.cc:2802] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:58.683260 9454 raft_consensus.cc:695] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Becoming Leader. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Running, Role: LEADER
I20250627 13:53:58.683545 9454 consensus_queue.cc:237] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } }
I20250627 13:53:58.686316 7655 catalog_manager.cc:5582] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83574c1bf8ea45adaff939158f93c096 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } }
I20250627 13:53:58.904511 9180 consensus_queue.cc:1035] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:58.926621 9492 consensus_queue.cc:1035] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250627 13:53:59.090744 9490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:53:59.091022 9490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:53:59.098271 9490 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20250627 13:53:59.167584 9444 consensus_queue.cc:1035] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:59.181964 9369 consensus_queue.cc:1035] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:59.190088 9454 consensus_queue.cc:1035] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:53:59.224140 9488 consensus_queue.cc:1035] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 323.716 ms
time per row: 0.161858 ms
I20250627 13:53:59.561388 7654 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45288:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250627 13:53:59.570428 7911 tablet_service.cc:1468] Processing CreateTablet for tablet 83cfc975906745a09c5e1dcfac812ae2 (DEFAULT_TABLE table=rebalance_test_table_9 [id=19e5686b50fe4187a733aa6413ef2ce0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:59.570650 8041 tablet_service.cc:1468] Processing CreateTablet for tablet 83cfc975906745a09c5e1dcfac812ae2 (DEFAULT_TABLE table=rebalance_test_table_9 [id=19e5686b50fe4187a733aa6413ef2ce0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:59.570926 7911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83cfc975906745a09c5e1dcfac812ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:59.571034 8041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83cfc975906745a09c5e1dcfac812ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:59.571197 8171 tablet_service.cc:1468] Processing CreateTablet for tablet 96fe0309adc54fb6a426bae02f26f898 (DEFAULT_TABLE table=rebalance_test_table_9 [id=19e5686b50fe4187a733aa6413ef2ce0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:59.571554 8171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96fe0309adc54fb6a426bae02f26f898. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:59.572139 7912 tablet_service.cc:1468] Processing CreateTablet for tablet 96fe0309adc54fb6a426bae02f26f898 (DEFAULT_TABLE table=rebalance_test_table_9 [id=19e5686b50fe4187a733aa6413ef2ce0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:59.572492 7912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96fe0309adc54fb6a426bae02f26f898. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:59.574774 7782 tablet_service.cc:1468] Processing CreateTablet for tablet eed3236c4fe8455cb9527b9f4b754e81 (DEFAULT_TABLE table=rebalance_test_table_9 [id=19e5686b50fe4187a733aa6413ef2ce0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:59.575141 7782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eed3236c4fe8455cb9527b9f4b754e81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:59.575462 8172 tablet_service.cc:1468] Processing CreateTablet for tablet 83cfc975906745a09c5e1dcfac812ae2 (DEFAULT_TABLE table=rebalance_test_table_9 [id=19e5686b50fe4187a733aa6413ef2ce0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:59.575851 8172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83cfc975906745a09c5e1dcfac812ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:59.576143 7913 tablet_service.cc:1468] Processing CreateTablet for tablet eed3236c4fe8455cb9527b9f4b754e81 (DEFAULT_TABLE table=rebalance_test_table_9 [id=19e5686b50fe4187a733aa6413ef2ce0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:59.576494 7913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eed3236c4fe8455cb9527b9f4b754e81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:59.578660 8043 tablet_service.cc:1468] Processing CreateTablet for tablet 96fe0309adc54fb6a426bae02f26f898 (DEFAULT_TABLE table=rebalance_test_table_9 [id=19e5686b50fe4187a733aa6413ef2ce0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:59.578986 8043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96fe0309adc54fb6a426bae02f26f898. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:59.582005 8042 tablet_service.cc:1468] Processing CreateTablet for tablet eed3236c4fe8455cb9527b9f4b754e81 (DEFAULT_TABLE table=rebalance_test_table_9 [id=19e5686b50fe4187a733aa6413ef2ce0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250627 13:53:59.582296 8042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eed3236c4fe8455cb9527b9f4b754e81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:53:59.594022 9554 tablet_bootstrap.cc:492] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:53:59.599308 9554 tablet_bootstrap.cc:654] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:59.601043 9555 tablet_bootstrap.cc:492] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap starting.
I20250627 13:53:59.601697 9553 tablet_bootstrap.cc:492] T 96fe0309adc54fb6a426bae02f26f898 P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
I20250627 13:53:59.603021 9554 tablet_bootstrap.cc:492] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854: No bootstrap required, opened a new log
I20250627 13:53:59.603163 9554 ts_tablet_manager.cc:1397] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20250627 13:53:59.603753 9554 raft_consensus.cc:357] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:59.603904 9554 raft_consensus.cc:383] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:59.603942 9553 tablet_bootstrap.cc:654] T 96fe0309adc54fb6a426bae02f26f898 P e4c831df4da841929d13d6c783099b0e: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:59.604038 9554 raft_consensus.cc:738] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: FOLLOWER
I20250627 13:53:59.604249 9554 consensus_queue.cc:260] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:59.604817 9554 ts_tablet_manager.cc:1428] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:53:59.605113 9554 tablet_bootstrap.cc:492] T eed3236c4fe8455cb9527b9f4b754e81 P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:53:59.606240 9555 tablet_bootstrap.cc:654] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:59.607046 9554 tablet_bootstrap.cc:654] T eed3236c4fe8455cb9527b9f4b754e81 P a41a89ae8ed743edb2b5cf4490d8c854: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:59.615721 9556 tablet_bootstrap.cc:492] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:53:59.616341 9555 tablet_bootstrap.cc:492] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660: No bootstrap required, opened a new log
I20250627 13:53:59.616542 9555 ts_tablet_manager.cc:1397] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.003s
I20250627 13:53:59.617538 9556 tablet_bootstrap.cc:654] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:59.617471 9555 raft_consensus.cc:357] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:59.617816 9555 raft_consensus.cc:383] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:59.617978 9555 raft_consensus.cc:738] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Initialized, Role: FOLLOWER
I20250627 13:53:59.618283 9555 consensus_queue.cc:260] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:59.619019 9555 ts_tablet_manager.cc:1428] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:53:59.619351 9555 tablet_bootstrap.cc:492] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap starting.
I20250627 13:53:59.619838 9554 tablet_bootstrap.cc:492] T eed3236c4fe8455cb9527b9f4b754e81 P a41a89ae8ed743edb2b5cf4490d8c854: No bootstrap required, opened a new log
I20250627 13:53:59.619977 9553 tablet_bootstrap.cc:492] T 96fe0309adc54fb6a426bae02f26f898 P e4c831df4da841929d13d6c783099b0e: No bootstrap required, opened a new log
I20250627 13:53:59.620091 9554 ts_tablet_manager.cc:1397] T eed3236c4fe8455cb9527b9f4b754e81 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20250627 13:53:59.620142 9553 ts_tablet_manager.cc:1397] T 96fe0309adc54fb6a426bae02f26f898 P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.008s
I20250627 13:53:59.621034 9555 tablet_bootstrap.cc:654] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:59.620981 9553 raft_consensus.cc:357] T 96fe0309adc54fb6a426bae02f26f898 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:59.621009 9554 raft_consensus.cc:357] T eed3236c4fe8455cb9527b9f4b754e81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:59.621351 9554 raft_consensus.cc:383] T eed3236c4fe8455cb9527b9f4b754e81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:59.621475 9554 raft_consensus.cc:738] T eed3236c4fe8455cb9527b9f4b754e81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: FOLLOWER
I20250627 13:53:59.621852 9554 consensus_queue.cc:260] T eed3236c4fe8455cb9527b9f4b754e81 P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:59.621181 9553 raft_consensus.cc:383] T 96fe0309adc54fb6a426bae02f26f898 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:59.622275 9553 raft_consensus.cc:738] T 96fe0309adc54fb6a426bae02f26f898 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: FOLLOWER
I20250627 13:53:59.622570 9554 ts_tablet_manager.cc:1428] T eed3236c4fe8455cb9527b9f4b754e81 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250627 13:53:59.622579 9553 consensus_queue.cc:260] T 96fe0309adc54fb6a426bae02f26f898 P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:59.622953 9554 tablet_bootstrap.cc:492] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:53:59.623732 9553 ts_tablet_manager.cc:1428] T 96fe0309adc54fb6a426bae02f26f898 P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250627 13:53:59.624086 9553 tablet_bootstrap.cc:492] T 83cfc975906745a09c5e1dcfac812ae2 P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
I20250627 13:53:59.625402 9554 tablet_bootstrap.cc:654] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:59.626607 9553 tablet_bootstrap.cc:654] T 83cfc975906745a09c5e1dcfac812ae2 P e4c831df4da841929d13d6c783099b0e: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:59.631557 9553 tablet_bootstrap.cc:492] T 83cfc975906745a09c5e1dcfac812ae2 P e4c831df4da841929d13d6c783099b0e: No bootstrap required, opened a new log
I20250627 13:53:59.632113 9553 ts_tablet_manager.cc:1397] T 83cfc975906745a09c5e1dcfac812ae2 P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20250627 13:53:59.633194 9553 raft_consensus.cc:357] T 83cfc975906745a09c5e1dcfac812ae2 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:59.634508 9553 raft_consensus.cc:383] T 83cfc975906745a09c5e1dcfac812ae2 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:59.635213 9553 raft_consensus.cc:738] T 83cfc975906745a09c5e1dcfac812ae2 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: FOLLOWER
I20250627 13:53:59.635936 9553 consensus_queue.cc:260] T 83cfc975906745a09c5e1dcfac812ae2 P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:59.636941 9553 ts_tablet_manager.cc:1428] T 83cfc975906745a09c5e1dcfac812ae2 P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250627 13:53:59.637836 9554 tablet_bootstrap.cc:492] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854: No bootstrap required, opened a new log
I20250627 13:53:59.638064 9554 ts_tablet_manager.cc:1397] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.004s
I20250627 13:53:59.638947 9554 raft_consensus.cc:357] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:59.639148 9554 raft_consensus.cc:383] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:59.639312 9554 raft_consensus.cc:738] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: FOLLOWER
I20250627 13:53:59.639604 9554 consensus_queue.cc:260] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:59.640269 9554 ts_tablet_manager.cc:1428] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:53:59.646916 9556 tablet_bootstrap.cc:492] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096: No bootstrap required, opened a new log
I20250627 13:53:59.647066 9556 ts_tablet_manager.cc:1397] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.009s
I20250627 13:53:59.647074 9555 tablet_bootstrap.cc:492] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660: No bootstrap required, opened a new log
I20250627 13:53:59.647235 9555 ts_tablet_manager.cc:1397] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent bootstrapping tablet: real 0.028s user 0.022s sys 0.004s
I20250627 13:53:59.647809 9556 raft_consensus.cc:357] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:59.647962 9556 raft_consensus.cc:383] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:59.648064 9556 raft_consensus.cc:738] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: FOLLOWER
I20250627 13:53:59.648077 9555 raft_consensus.cc:357] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:59.648280 9555 raft_consensus.cc:383] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:59.648401 9555 raft_consensus.cc:738] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Initialized, Role: FOLLOWER
I20250627 13:53:59.648299 9556 consensus_queue.cc:260] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:59.648669 9555 consensus_queue.cc:260] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:59.648850 9556 ts_tablet_manager.cc:1428] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250627 13:53:59.649232 9555 ts_tablet_manager.cc:1428] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:53:59.649523 9555 tablet_bootstrap.cc:492] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap starting.
I20250627 13:53:59.651199 9555 tablet_bootstrap.cc:654] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660: Neither blocks nor log segments found. Creating new log.
I20250627 13:53:59.653712 9555 tablet_bootstrap.cc:492] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660: No bootstrap required, opened a new log
I20250627 13:53:59.653901 9555 ts_tablet_manager.cc:1397] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20250627 13:53:59.654582 9555 raft_consensus.cc:357] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:59.654726 9555 raft_consensus.cc:383] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:53:59.654794 9555 raft_consensus.cc:738] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Initialized, Role: FOLLOWER
I20250627 13:53:59.654999 9555 consensus_queue.cc:260] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:59.655439 9555 ts_tablet_manager.cc:1428] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250627 13:53:59.673095 9538 raft_consensus.cc:491] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:59.673262 9538 raft_consensus.cc:513] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:59.673849 9538 leader_election.cc:290] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675), e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:53:59.674017 9492 raft_consensus.cc:491] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:59.674221 9492 raft_consensus.cc:513] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:59.674312 7932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96fe0309adc54fb6a426bae02f26f898" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" is_pre_election: true
I20250627 13:53:59.674346 8193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96fe0309adc54fb6a426bae02f26f898" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e" is_pre_election: true
I20250627 13:53:59.674556 8193 raft_consensus.cc:2466] T 96fe0309adc54fb6a426bae02f26f898 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 0.
I20250627 13:53:59.674584 7932 raft_consensus.cc:2466] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 0.
I20250627 13:53:59.674918 9492 leader_election.cc:290] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:53:59.674995 7998 leader_election.cc:304] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cf479bdcd0242738d2a77db4ad9f660, a41a89ae8ed743edb2b5cf4490d8c854; no voters:
I20250627 13:53:59.675390 9538 raft_consensus.cc:2802] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:59.675436 8193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96fe0309adc54fb6a426bae02f26f898" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e" is_pre_election: true
I20250627 13:53:59.675521 9538 raft_consensus.cc:491] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:59.675618 9538 raft_consensus.cc:3058] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:59.675623 8193 raft_consensus.cc:2466] T 96fe0309adc54fb6a426bae02f26f898 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a41a89ae8ed743edb2b5cf4490d8c854 in term 0.
I20250627 13:53:59.675626 8062 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96fe0309adc54fb6a426bae02f26f898" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660" is_pre_election: true
I20250627 13:53:59.676189 7869 leader_election.cc:304] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a41a89ae8ed743edb2b5cf4490d8c854, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:59.676468 9492 raft_consensus.cc:2802] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:59.676561 9492 raft_consensus.cc:491] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:59.676661 9492 raft_consensus.cc:3058] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:59.678136 9538 raft_consensus.cc:513] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:59.678480 8062 raft_consensus.cc:2391] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a41a89ae8ed743edb2b5cf4490d8c854 in current term 1: Already voted for candidate 9cf479bdcd0242738d2a77db4ad9f660 in this term.
I20250627 13:53:59.678743 9492 raft_consensus.cc:513] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:59.679028 7932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96fe0309adc54fb6a426bae02f26f898" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
I20250627 13:53:59.679275 7932 raft_consensus.cc:2391] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9cf479bdcd0242738d2a77db4ad9f660 in current term 1: Already voted for candidate a41a89ae8ed743edb2b5cf4490d8c854 in this term.
I20250627 13:53:59.679314 8193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96fe0309adc54fb6a426bae02f26f898" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e"
I20250627 13:53:59.679358 9492 leader_election.cc:290] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 1 election: Requested vote from peers e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:53:59.679455 8193 raft_consensus.cc:3058] T 96fe0309adc54fb6a426bae02f26f898 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:59.679831 8062 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96fe0309adc54fb6a426bae02f26f898" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660"
I20250627 13:53:59.680071 8062 raft_consensus.cc:2391] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a41a89ae8ed743edb2b5cf4490d8c854 in current term 1: Already voted for candidate 9cf479bdcd0242738d2a77db4ad9f660 in this term.
I20250627 13:53:59.680065 8192 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96fe0309adc54fb6a426bae02f26f898" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e"
I20250627 13:53:59.680212 9538 leader_election.cc:290] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 election: Requested vote from peers a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675), e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:53:59.680670 7866 leader_election.cc:304] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a41a89ae8ed743edb2b5cf4490d8c854; no voters: 9cf479bdcd0242738d2a77db4ad9f660, e4c831df4da841929d13d6c783099b0e
I20250627 13:53:59.680929 9492 raft_consensus.cc:2747] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250627 13:53:59.681901 8193 raft_consensus.cc:2466] T 96fe0309adc54fb6a426bae02f26f898 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 1.
I20250627 13:53:59.682274 7999 leader_election.cc:304] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9cf479bdcd0242738d2a77db4ad9f660, e4c831df4da841929d13d6c783099b0e; no voters: a41a89ae8ed743edb2b5cf4490d8c854
I20250627 13:53:59.682514 9444 raft_consensus.cc:2802] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:59.682651 9444 raft_consensus.cc:695] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Becoming Leader. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Running, Role: LEADER
I20250627 13:53:59.682907 9444 consensus_queue.cc:237] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:59.685590 7658 catalog_manager.cc:5582] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } }
I20250627 13:53:59.774210 9444 raft_consensus.cc:491] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:59.774425 9444 raft_consensus.cc:513] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:59.775033 9444 leader_election.cc:290] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675), 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:53:59.775437 7932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eed3236c4fe8455cb9527b9f4b754e81" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" is_pre_election: true
I20250627 13:53:59.775486 7800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eed3236c4fe8455cb9527b9f4b754e81" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096" is_pre_election: true
I20250627 13:53:59.775681 7932 raft_consensus.cc:2466] T eed3236c4fe8455cb9527b9f4b754e81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 0.
I20250627 13:53:59.775692 7800 raft_consensus.cc:2466] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 0.
I20250627 13:53:59.776101 7998 leader_election.cc:304] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cf479bdcd0242738d2a77db4ad9f660, a41a89ae8ed743edb2b5cf4490d8c854; no voters:
I20250627 13:53:59.776392 9444 raft_consensus.cc:2802] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:59.776492 9444 raft_consensus.cc:491] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:59.776597 9444 raft_consensus.cc:3058] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:59.779610 9444 raft_consensus.cc:513] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:59.780191 9444 leader_election.cc:290] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 election: Requested vote from peers a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675), 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:53:59.780748 7800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eed3236c4fe8455cb9527b9f4b754e81" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83574c1bf8ea45adaff939158f93c096"
I20250627 13:53:59.780946 7800 raft_consensus.cc:3058] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:59.780930 7932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eed3236c4fe8455cb9527b9f4b754e81" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
I20250627 13:53:59.781143 7932 raft_consensus.cc:3058] T eed3236c4fe8455cb9527b9f4b754e81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:59.783949 7932 raft_consensus.cc:2466] T eed3236c4fe8455cb9527b9f4b754e81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 1.
I20250627 13:53:59.783928 7800 raft_consensus.cc:2466] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 1.
I20250627 13:53:59.784541 7996 leader_election.cc:304] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, 9cf479bdcd0242738d2a77db4ad9f660; no voters:
I20250627 13:53:59.784828 9444 raft_consensus.cc:2802] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:59.785008 9444 raft_consensus.cc:695] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Becoming Leader. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Running, Role: LEADER
I20250627 13:53:59.785279 9444 consensus_queue.cc:237] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } }
I20250627 13:53:59.788640 7658 catalog_manager.cc:5582] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } }
I20250627 13:53:59.942728 9444 raft_consensus.cc:491] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:53:59.942891 9444 raft_consensus.cc:513] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:59.943416 9444 leader_election.cc:290] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675), e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:53:59.943888 8193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "83cfc975906745a09c5e1dcfac812ae2" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e" is_pre_election: true
I20250627 13:53:59.943905 7932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "83cfc975906745a09c5e1dcfac812ae2" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" is_pre_election: true
I20250627 13:53:59.944126 8193 raft_consensus.cc:2466] T 83cfc975906745a09c5e1dcfac812ae2 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 0.
I20250627 13:53:59.944130 7932 raft_consensus.cc:2466] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 0.
I20250627 13:53:59.944504 7999 leader_election.cc:304] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cf479bdcd0242738d2a77db4ad9f660, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:59.944777 9444 raft_consensus.cc:2802] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:53:59.944895 9444 raft_consensus.cc:491] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:53:59.944973 9444 raft_consensus.cc:3058] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:59.947055 9444 raft_consensus.cc:513] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:59.947625 9444 leader_election.cc:290] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 election: Requested vote from peers a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675), e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:53:59.947899 7933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "83cfc975906745a09c5e1dcfac812ae2" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
I20250627 13:53:59.948096 7933 raft_consensus.cc:3058] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:59.948107 8193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "83cfc975906745a09c5e1dcfac812ae2" candidate_uuid: "9cf479bdcd0242738d2a77db4ad9f660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e"
I20250627 13:53:59.948298 8193 raft_consensus.cc:3058] T 83cfc975906745a09c5e1dcfac812ae2 P e4c831df4da841929d13d6c783099b0e [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:53:59.951110 7933 raft_consensus.cc:2466] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 1.
I20250627 13:53:59.951311 8193 raft_consensus.cc:2466] T 83cfc975906745a09c5e1dcfac812ae2 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf479bdcd0242738d2a77db4ad9f660 in term 1.
I20250627 13:53:59.951720 7999 leader_election.cc:304] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cf479bdcd0242738d2a77db4ad9f660, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:53:59.952006 9444 raft_consensus.cc:2802] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:53:59.952193 9444 raft_consensus.cc:695] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Becoming Leader. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Running, Role: LEADER
I20250627 13:53:59.952502 9444 consensus_queue.cc:237] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } }
I20250627 13:53:59.955170 7658 catalog_manager.cc:5582] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:00.108983 9444 consensus_queue.cc:1035] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:54:00.113584 9538 consensus_queue.cc:1035] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:54:00.181035 9538 consensus_queue.cc:1035] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:54:00.187999 9538 consensus_queue.cc:1035] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:54:00.363484 9538 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:54:00.371304 9572 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250627 13:54:00.407752 9570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:54:00.408066 9570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:54:00.415714 9570 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
INSERT report
rows total: 2000
time total: 217.037 ms
time per row: 0.108518 ms
I20250627 13:54:00.784150 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.69:33101
--local_ip_for_outbound_sockets=127.0.103.69
--tserver_master_addrs=127.0.103.126:35849
--webserver_port=35087
--webserver_interface=127.0.103.69
--builtin_ntp_servers=127.0.103.84:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250627 13:54:01.106732 9629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:54:01.107081 9629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:54:01.107211 9629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:54:01.114778 9629 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:54:01.114878 9629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:54:01.115002 9629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.69
I20250627 13:54:01.125187 9629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.69:33101
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.103.69
--webserver_port=35087
--tserver_master_addrs=127.0.103.126:35849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.69
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:54:01.126775 9629 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:54:01.128536 9629 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:54:01.139890 9635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:01.139855 9634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:01.140643 9637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:54:01.140908 9629 server_base.cc:1048] running on GCE node
I20250627 13:54:01.141430 9629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:54:01.142355 9629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:54:01.143561 9629 hybrid_clock.cc:648] HybridClock initialized: now 1751032441143507 us; error 32 us; skew 500 ppm
I20250627 13:54:01.143824 9629 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:54:01.146231 9629 webserver.cc:469] Webserver started at http://127.0.103.69:35087/ using document root <none> and password file <none>
I20250627 13:54:01.146986 9629 fs_manager.cc:362] Metadata directory not provided
I20250627 13:54:01.147114 9629 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:54:01.152694 9629 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250627 13:54:01.155632 9643 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:01.157577 9629 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250627 13:54:01.157773 9629 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "2a1282ebe92a4c9a92eed3b13f69012a"
format_stamp: "Formatted at 2025-06-27 13:53:42 on dist-test-slave-1ckg"
I20250627 13:54:01.158485 9629 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:54:01.180052 9629 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:54:01.181135 9629 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:54:01.181442 9629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:54:01.182473 9629 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:54:01.184031 9629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:54:01.184114 9629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:01.184188 9629 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:54:01.184237 9629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:01.222573 9629 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.69:33101
I20250627 13:54:01.222640 9755 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.69:33101 every 8 connection(s)
I20250627 13:54:01.223933 9629 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250627 13:54:01.230232 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 9629
I20250627 13:54:01.230933 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.70:40647
--local_ip_for_outbound_sockets=127.0.103.70
--tserver_master_addrs=127.0.103.126:35849
--webserver_port=41307
--webserver_interface=127.0.103.70
--builtin_ntp_servers=127.0.103.84:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:54:01.237038 9756 heartbeater.cc:344] Connected to a master server at 127.0.103.126:35849
I20250627 13:54:01.237411 9756 heartbeater.cc:461] Registering TS with master...
I20250627 13:54:01.238505 9756 heartbeater.cc:507] Master 127.0.103.126:35849 requested a full tablet report, sending...
I20250627 13:54:01.240195 7659 ts_manager.cc:194] Re-registered known tserver with Master: 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101)
I20250627 13:54:01.241127 7659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.69:55897
W20250627 13:54:01.565766 9760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:54:01.566035 9760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:54:01.566166 9760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:54:01.573325 9760 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:54:01.573403 9760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:54:01.573562 9760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.70
I20250627 13:54:01.582363 9760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.70:40647
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.103.70
--webserver_port=41307
--tserver_master_addrs=127.0.103.126:35849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.70
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:54:01.583765 9760 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:54:01.585932 9760 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:54:01.597003 9768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:01.597039 9765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:01.597419 9766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:54:01.603242 9760 server_base.cc:1048] running on GCE node
I20250627 13:54:01.603797 9760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:54:01.604624 9760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:54:01.605844 9760 hybrid_clock.cc:648] HybridClock initialized: now 1751032441605783 us; error 35 us; skew 500 ppm
I20250627 13:54:01.606107 9760 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:54:01.608534 9760 webserver.cc:469] Webserver started at http://127.0.103.70:41307/ using document root <none> and password file <none>
I20250627 13:54:01.609225 9760 fs_manager.cc:362] Metadata directory not provided
I20250627 13:54:01.609361 9760 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:54:01.614426 9760 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20250627 13:54:01.616923 9774 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:01.618124 9760 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250627 13:54:01.618273 9760 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "956ff058e6d64579a1963aabdb6d466b"
format_stamp: "Formatted at 2025-06-27 13:53:42 on dist-test-slave-1ckg"
I20250627 13:54:01.618906 9760 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:54:01.640914 9760 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:54:01.641813 9760 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:54:01.642046 9760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:54:01.642984 9760 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:54:01.644385 9760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:54:01.644480 9760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:01.644606 9760 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:54:01.644680 9760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:01.678588 9760 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.70:40647
I20250627 13:54:01.678678 9887 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.70:40647 every 8 connection(s)
I20250627 13:54:01.679967 9760 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20250627 13:54:01.689198 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 9760
I20250627 13:54:01.689987 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.71:37859
--local_ip_for_outbound_sockets=127.0.103.71
--tserver_master_addrs=127.0.103.126:35849
--webserver_port=36169
--webserver_interface=127.0.103.71
--builtin_ntp_servers=127.0.103.84:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:54:01.691987 9888 heartbeater.cc:344] Connected to a master server at 127.0.103.126:35849
I20250627 13:54:01.692305 9888 heartbeater.cc:461] Registering TS with master...
I20250627 13:54:01.693176 9888 heartbeater.cc:507] Master 127.0.103.126:35849 requested a full tablet report, sending...
I20250627 13:54:01.694728 7659 ts_manager.cc:194] Re-registered known tserver with Master: 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647)
I20250627 13:54:01.695730 7659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.70:38925
W20250627 13:54:02.011397 9891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:54:02.011674 9891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:54:02.011759 9891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:54:02.019215 9891 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:54:02.019371 9891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:54:02.019568 9891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.71
I20250627 13:54:02.029476 9891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.71:37859
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.103.71
--webserver_port=36169
--tserver_master_addrs=127.0.103.126:35849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.71
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:54:02.031150 9891 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:54:02.032867 9891 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:54:02.043933 9900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:02.043928 9898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:02.043956 9897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:54:02.044816 9891 server_base.cc:1048] running on GCE node
I20250627 13:54:02.045337 9891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:54:02.046276 9891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:54:02.047510 9891 hybrid_clock.cc:648] HybridClock initialized: now 1751032442047453 us; error 35 us; skew 500 ppm
I20250627 13:54:02.047753 9891 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:54:02.049924 9891 webserver.cc:469] Webserver started at http://127.0.103.71:36169/ using document root <none> and password file <none>
I20250627 13:54:02.050540 9891 fs_manager.cc:362] Metadata directory not provided
I20250627 13:54:02.050647 9891 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:54:02.055445 9891 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20250627 13:54:02.058115 9906 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:02.059281 9891 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250627 13:54:02.059437 9891 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "6686c6c365bf4f4582058890161676fe"
format_stamp: "Formatted at 2025-06-27 13:53:43 on dist-test-slave-1ckg"
I20250627 13:54:02.060199 9891 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:54:02.084344 9891 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:54:02.085351 9891 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:54:02.085615 9891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:54:02.086544 9891 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:54:02.088183 9891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:54:02.088277 9891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:02.088403 9891 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:54:02.088470 9891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:02.121351 9891 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.71:37859
I20250627 13:54:02.121416 10018 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.71:37859 every 8 connection(s)
I20250627 13:54:02.122835 9891 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20250627 13:54:02.126245 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 9891
I20250627 13:54:02.127008 413 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
/tmp/dist-test-taskNzQG6V/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.103.72:35213
--local_ip_for_outbound_sockets=127.0.103.72
--tserver_master_addrs=127.0.103.126:35849
--webserver_port=43603
--webserver_interface=127.0.103.72
--builtin_ntp_servers=127.0.103.84:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250627 13:54:02.136747 10019 heartbeater.cc:344] Connected to a master server at 127.0.103.126:35849
I20250627 13:54:02.137163 10019 heartbeater.cc:461] Registering TS with master...
I20250627 13:54:02.138235 10019 heartbeater.cc:507] Master 127.0.103.126:35849 requested a full tablet report, sending...
I20250627 13:54:02.139976 7659 ts_manager.cc:194] Re-registered known tserver with Master: 6686c6c365bf4f4582058890161676fe (127.0.103.71:37859)
I20250627 13:54:02.140750 7659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.71:34417
I20250627 13:54:02.243530 9756 heartbeater.cc:499] Master 127.0.103.126:35849 was elected leader, sending a full tablet report...
W20250627 13:54:02.466039 10022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:54:02.466421 10022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:54:02.466567 10022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:54:02.475526 10022 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250627 13:54:02.475664 10022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:54:02.475967 10022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.103.72
I20250627 13:54:02.485430 10022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.103.84:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.103.72:35213
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.103.72
--webserver_port=43603
--tserver_master_addrs=127.0.103.126:35849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.103.72
--log_dir=/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:54:02.487332 10022 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:54:02.489153 10022 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:54:02.499861 10028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:02.499815 10027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:02.500391 10030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:54:02.500813 10022 server_base.cc:1048] running on GCE node
I20250627 13:54:02.501366 10022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:54:02.502220 10022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:54:02.503422 10022 hybrid_clock.cc:648] HybridClock initialized: now 1751032442503359 us; error 40 us; skew 500 ppm
I20250627 13:54:02.503665 10022 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:54:02.505985 10022 webserver.cc:469] Webserver started at http://127.0.103.72:43603/ using document root <none> and password file <none>
I20250627 13:54:02.506835 10022 fs_manager.cc:362] Metadata directory not provided
I20250627 13:54:02.506973 10022 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:54:02.511977 10022 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20250627 13:54:02.514333 10036 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:02.515563 10022 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250627 13:54:02.515731 10022 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "dede81a8c0164427a61532d01b6fc157"
format_stamp: "Formatted at 2025-06-27 13:53:43 on dist-test-slave-1ckg"
I20250627 13:54:02.516391 10022 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:54:02.539534 10022 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:54:02.540321 10022 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:54:02.540573 10022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:54:02.541437 10022 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:54:02.543155 10022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:54:02.543277 10022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:02.543385 10022 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:54:02.543458 10022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:02.578691 10022 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.72:35213
I20250627 13:54:02.578778 10148 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.72:35213 every 8 connection(s)
I20250627 13:54:02.580163 10022 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20250627 13:54:02.587257 413 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskNzQG6V/build/asan/bin/kudu as pid 10022
I20250627 13:54:02.593572 10149 heartbeater.cc:344] Connected to a master server at 127.0.103.126:35849
I20250627 13:54:02.593957 10149 heartbeater.cc:461] Registering TS with master...
I20250627 13:54:02.594906 10149 heartbeater.cc:507] Master 127.0.103.126:35849 requested a full tablet report, sending...
I20250627 13:54:02.596305 7659 ts_manager.cc:194] Re-registered known tserver with Master: dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213)
I20250627 13:54:02.597079 7659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.103.72:38389
I20250627 13:54:02.698311 9888 heartbeater.cc:499] Master 127.0.103.126:35849 was elected leader, sending a full tablet report...
I20250627 13:54:03.143138 10019 heartbeater.cc:499] Master 127.0.103.126:35849 was elected leader, sending a full tablet report...
I20250627 13:54:03.600839 10149 heartbeater.cc:499] Master 127.0.103.126:35849 was elected leader, sending a full tablet report...
I20250627 13:54:03.869124 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:03.886658 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:03.897313 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:03.914490 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:03.926965 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:03.940238 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:03.964841 10084 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:03.994277 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:04.056939 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:04.070719 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:04.095510 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:04.167670 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:04.220027 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:04.234767 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:04.285079 10084 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:04.290596 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:04.295531 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:04.303958 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:04.311003 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:04.351773 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:04.365052 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:04.410507 10084 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:04.456557 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:04.505813 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:04.631224 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:04.652740 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:04.671720 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:04.674672 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:04.744797 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:04.758731 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:04.796772 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:04.818289 10084 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:04.824021 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:54:04.844368 7999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.68:46139, user_credentials={real_user=slave}} blocked reactor thread for 51425.8us
I20250627 13:54:04.851487 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:04.853415 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:04.895818 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:04.897665 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:04.923444 10084 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:04.987947 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:05.074420 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:05.496583 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:54:05.572136 7998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.66:43675, user_credentials={real_user=slave}} blocked reactor thread for 74898.7us
I20250627 13:54:05.578590 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:05.646096 10084 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:05.648054 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:05.657698 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:05.665423 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:05.692546 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:05.693285 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:06.069190 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:06.078063 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:06.151322 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:06.178076 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:06.197933 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:06.220417 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:06.222402 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:06.240097 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:06.240723 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:06.248939 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:06.265527 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:06.267002 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:06.279330 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:06.300854 10084 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:06.360227 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:06.382233 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:06.383543 10084 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:54:06.394788 7866 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.67:38933, user_credentials={real_user=slave}} blocked reactor thread for 58034.1us
I20250627 13:54:06.423708 10084 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:06.453590 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:06.462795 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:06.475315 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:06.504751 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:06.507444 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:06.530699 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:06.564324 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:06.604606 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:06.617383 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:06.628118 10084 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:06.641094 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:06.656826 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:06.678407 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:06.685015 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:07.210124 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:07.244638 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:07.272135 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:07.274533 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:07.279551 10084 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:07.327347 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:07.359886 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:07.441463 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
W20250627 13:54:07.616819 8128 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.66:43675, user_credentials={real_user=slave}} blocked reactor thread for 59873.2us
I20250627 13:54:07.646593 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:07.656926 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:07.681684 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:07.687707 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:07.695035 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:07.746999 10084 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:07.749277 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:07.776098 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:07.818296 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:07.836812 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:07.870469 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:07.888581 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:07.890497 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:07.917822 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:07.924767 10084 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:07.976755 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:08.181381 8061 consensus_queue.cc:237] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:08.184911 8191 consensus_queue.cc:237] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:08.226162 7801 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:08.226811 10713 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250627 13:54:08.243289 8129 consensus_peers.cc:487] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e -> Peer 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101): Couldn't send request to peer 2a1282ebe92a4c9a92eed3b13f69012a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e3053b529a264e8697a1df9e17eeea81. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:08.249118 7931 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:08.255004 10713 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:08.258035 7800 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:08.258726 10583 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:08.265007 10506 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:08.272363 8061 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:08.272554 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:54:08.273052 7998 consensus_peers.cc:487] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213): Couldn't send request to peer dede81a8c0164427a61532d01b6fc157. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2eb537d49804e1b8ec5a217b1564988. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:08.273008 10583 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:08.283671 7801 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:08.299396 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:08.297326 7933 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:08.306859 7653 catalog_manager.cc:5582] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:08.317149 10869 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:08.322418 8061 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:08.340135 8061 consensus_queue.cc:237] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:08.343911 7931 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:08.345479 10713 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250627 13:54:08.346644 7998 consensus_peers.cc:487] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213): Couldn't send request to peer dede81a8c0164427a61532d01b6fc157. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2eb537d49804e1b8ec5a217b1564988. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:08.348435 7800 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:08.349038 10713 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:08.352070 10713 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:08.354782 7931 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:08.356738 7800 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:08.378203 7652 catalog_manager.cc:5582] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index -1 to 4, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:08.403507 7800 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:08.404914 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:08.443413 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:08.445405 10084 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:08.466928 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:08.467657 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:08.475734 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:08.493261 7658 catalog_manager.cc:5582] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index 4 to 5, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
W20250627 13:54:08.500881 7996 consensus_peers.cc:487] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2eb537d49804e1b8ec5a217b1564988. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:08.513752 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:08.515671 10084 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:08.532341 7800 consensus_queue.cc:237] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:08.542384 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:08.555680 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:08.565824 8061 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:08.566370 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:08.566524 10759 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:08.568081 7931 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:08.573004 10759 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:08.574717 8063 consensus_queue.cc:237] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:08.576125 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:08.595496 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
W20250627 13:54:08.615463 7998 consensus_peers.cc:487] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213): Couldn't send request to peer dede81a8c0164427a61532d01b6fc157. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 644985e3086d4614b6598a727cb60c1d. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:08.615892 8191 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:08.616647 10887 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:08.631240 10928 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:08.637473 8062 consensus_queue.cc:237] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:08.648466 7933 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:08.652945 10506 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:08.664222 8060 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:08.670941 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:08.674223 7656 catalog_manager.cc:5582] T 87726824b05a4b968386f57d3a7a4aad P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index -1 to 4, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:08.677271 7800 raft_consensus.cc:1273] T 91664403eed74badbeba83d69f597e1f P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:08.677920 10713 consensus_queue.cc:1035] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:08.685940 10713 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
W20250627 13:54:08.687383 7738 consensus_peers.cc:487] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 -> Peer 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101): Couldn't send request to peer 2a1282ebe92a4c9a92eed3b13f69012a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87726824b05a4b968386f57d3a7a4aad. This is attempt 1: this message will repeat every 5th retry.
W20250627 13:54:08.690536 7998 consensus_peers.cc:487] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213): Couldn't send request to peer dede81a8c0164427a61532d01b6fc157. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91664403eed74badbeba83d69f597e1f. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:08.695039 7658 catalog_manager.cc:5582] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:08.698238 7931 raft_consensus.cc:1273] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:08.710562 10887 consensus_queue.cc:1035] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:08.712127 10887 raft_consensus.cc:2953] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:08.716861 7800 raft_consensus.cc:2953] T 91664403eed74badbeba83d69f597e1f P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:08.730358 7658 catalog_manager.cc:5582] T 91664403eed74badbeba83d69f597e1f P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:08.731295 8193 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:08.735078 7932 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:08.750061 7931 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:08.756134 7929 raft_consensus.cc:2953] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:08.777266 7801 consensus_queue.cc:237] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:08.812101 8062 consensus_queue.cc:237] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:08.827266 8193 raft_consensus.cc:1273] T 720cd645f0b7414f819fcbff3fd262c3 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20250627 13:54:08.827555 7738 consensus_peers.cc:487] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 -> Peer 6686c6c365bf4f4582058890161676fe (127.0.103.71:37859): Couldn't send request to peer 6686c6c365bf4f4582058890161676fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 720cd645f0b7414f819fcbff3fd262c3. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:08.827932 10930 consensus_queue.cc:1035] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:08.834599 7931 raft_consensus.cc:1273] T 720cd645f0b7414f819fcbff3fd262c3 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:08.835145 10930 consensus_queue.cc:1035] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250627 13:54:08.838040 7996 consensus_peers.cc:487] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 570b134177314b31851951c411a9f4bf. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:08.838141 10953 ts_tablet_manager.cc:927] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a: Initiating tablet copy from peer e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:54:08.837904 10930 raft_consensus.cc:2953] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:08.839114 7929 raft_consensus.cc:1273] T 570b134177314b31851951c411a9f4bf P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:08.842526 8193 raft_consensus.cc:1273] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:08.843135 10933 consensus_queue.cc:1035] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:08.844594 10933 consensus_queue.cc:1035] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:08.847399 8191 raft_consensus.cc:2953] T 720cd645f0b7414f819fcbff3fd262c3 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:08.862023 10953 tablet_copy_client.cc:323] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.68:46139
I20250627 13:54:08.863226 7656 catalog_manager.cc:5582] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index -1 to 4, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:08.871033 7932 raft_consensus.cc:2953] T 720cd645f0b7414f819fcbff3fd262c3 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:08.893039 8213 tablet_copy_service.cc:140] P e4c831df4da841929d13d6c783099b0e: Received BeginTabletCopySession request for tablet e3053b529a264e8697a1df9e17eeea81 from peer 2a1282ebe92a4c9a92eed3b13f69012a ({username='slave'} at 127.0.103.69:43551)
I20250627 13:54:08.893422 8213 tablet_copy_service.cc:161] P e4c831df4da841929d13d6c783099b0e: Beginning new tablet copy session on tablet e3053b529a264e8697a1df9e17eeea81 from peer 2a1282ebe92a4c9a92eed3b13f69012a at {username='slave'} at 127.0.103.69:43551: session id = 2a1282ebe92a4c9a92eed3b13f69012a-e3053b529a264e8697a1df9e17eeea81
I20250627 13:54:08.894505 10961 ts_tablet_manager.cc:927] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:08.897432 10933 raft_consensus.cc:2953] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:08.898188 7931 raft_consensus.cc:2953] T 570b134177314b31851951c411a9f4bf P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:08.905022 7658 catalog_manager.cc:5582] T 570b134177314b31851951c411a9f4bf P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:08.910281 8193 raft_consensus.cc:2953] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:08.910645 8213 tablet_copy_source_session.cc:215] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:08.917207 10953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3053b529a264e8697a1df9e17eeea81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:08.930850 10961 tablet_copy_client.cc:323] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:08.941882 10953 tablet_copy_client.cc:806] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:08.942193 10953 tablet_copy_client.cc:670] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:08.944875 10953 tablet_copy_client.cc:538] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:08.949508 10953 tablet_bootstrap.cc:492] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap starting.
I20250627 13:54:08.965036 10953 log.cc:826] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a: Log is configured to *not* fsync() on all Append() calls
I20250627 13:54:08.965587 8083 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet d2eb537d49804e1b8ec5a217b1564988 from peer 956ff058e6d64579a1963aabdb6d466b ({username='slave'} at 127.0.103.70:59981)
I20250627 13:54:08.965883 8083 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet d2eb537d49804e1b8ec5a217b1564988 from peer 956ff058e6d64579a1963aabdb6d466b at {username='slave'} at 127.0.103.70:59981: session id = 956ff058e6d64579a1963aabdb6d466b-d2eb537d49804e1b8ec5a217b1564988
I20250627 13:54:08.970745 8083 tablet_copy_source_session.cc:215] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:08.975203 7801 consensus_queue.cc:237] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:08.982508 10961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2eb537d49804e1b8ec5a217b1564988. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:08.988801 10975 ts_tablet_manager.cc:927] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
W20250627 13:54:08.993011 7738 consensus_peers.cc:487] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 -> Peer 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101): Couldn't send request to peer 2a1282ebe92a4c9a92eed3b13f69012a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c322cfebf12c4fef8d88fbd68e7e8b6f. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:08.994283 8062 raft_consensus.cc:1273] T c322cfebf12c4fef8d88fbd68e7e8b6f P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:08.994987 10930 consensus_queue.cc:1035] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:09.001879 7931 raft_consensus.cc:1273] T c322cfebf12c4fef8d88fbd68e7e8b6f P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:09.002599 10930 consensus_queue.cc:1035] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:09.020105 10975 tablet_copy_client.cc:323] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:09.020252 10930 raft_consensus.cc:2953] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:09.029119 7658 catalog_manager.cc:5582] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index -1 to 4, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:09.040979 7931 raft_consensus.cc:2953] T c322cfebf12c4fef8d88fbd68e7e8b6f P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:09.026927 8062 raft_consensus.cc:2953] T c322cfebf12c4fef8d88fbd68e7e8b6f P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:09.083534 8083 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet d2eb537d49804e1b8ec5a217b1564988 from peer dede81a8c0164427a61532d01b6fc157 ({username='slave'} at 127.0.103.72:50607)
I20250627 13:54:09.083724 8083 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet d2eb537d49804e1b8ec5a217b1564988 from peer dede81a8c0164427a61532d01b6fc157 at {username='slave'} at 127.0.103.72:50607: session id = dede81a8c0164427a61532d01b6fc157-d2eb537d49804e1b8ec5a217b1564988
I20250627 13:54:09.087092 8083 tablet_copy_source_session.cc:215] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:09.094542 10953 tablet_bootstrap.cc:492] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:09.095666 10975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2eb537d49804e1b8ec5a217b1564988. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:09.095794 10953 tablet_bootstrap.cc:492] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap complete.
I20250627 13:54:09.096459 10953 ts_tablet_manager.cc:1397] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a: Time spent bootstrapping tablet: real 0.147s user 0.070s sys 0.017s
I20250627 13:54:09.103281 10953 raft_consensus.cc:357] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:09.103957 10953 raft_consensus.cc:738] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a1282ebe92a4c9a92eed3b13f69012a, State: Initialized, Role: LEARNER
I20250627 13:54:09.104820 10953 consensus_queue.cc:260] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:09.114219 10953 ts_tablet_manager.cc:1428] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a: Time spent starting tablet: real 0.018s user 0.008s sys 0.008s
I20250627 13:54:09.116798 8213 tablet_copy_service.cc:342] P e4c831df4da841929d13d6c783099b0e: Request end of tablet copy session 2a1282ebe92a4c9a92eed3b13f69012a-e3053b529a264e8697a1df9e17eeea81 received from {username='slave'} at 127.0.103.69:43551
I20250627 13:54:09.117022 8213 tablet_copy_service.cc:434] P e4c831df4da841929d13d6c783099b0e: ending tablet copy session 2a1282ebe92a4c9a92eed3b13f69012a-e3053b529a264e8697a1df9e17eeea81 on tablet e3053b529a264e8697a1df9e17eeea81 with peer 2a1282ebe92a4c9a92eed3b13f69012a
I20250627 13:54:09.119686 10988 ts_tablet_manager.cc:927] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:09.124953 10988 tablet_copy_client.cc:323] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:09.125424 10961 tablet_copy_client.cc:806] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:09.125965 8083 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet 644985e3086d4614b6598a727cb60c1d from peer dede81a8c0164427a61532d01b6fc157 ({username='slave'} at 127.0.103.72:50607)
I20250627 13:54:09.126135 8083 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet 644985e3086d4614b6598a727cb60c1d from peer dede81a8c0164427a61532d01b6fc157 at {username='slave'} at 127.0.103.72:50607: session id = dede81a8c0164427a61532d01b6fc157-644985e3086d4614b6598a727cb60c1d
I20250627 13:54:09.128916 8083 tablet_copy_source_session.cc:215] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:09.130481 10988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 644985e3086d4614b6598a727cb60c1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:09.131321 10975 tablet_copy_client.cc:806] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:09.131193 7801 consensus_queue.cc:237] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:09.131603 10975 tablet_copy_client.cc:670] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:09.138528 7931 consensus_queue.cc:237] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:09.139875 10975 tablet_copy_client.cc:538] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:09.142616 8193 raft_consensus.cc:1273] T ff83fdbab2b9417eac50495c54f0da54 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:09.143023 7929 raft_consensus.cc:1273] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20250627 13:54:09.143699 7738 consensus_peers.cc:487] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 -> Peer 6686c6c365bf4f4582058890161676fe (127.0.103.71:37859): Couldn't send request to peer 6686c6c365bf4f4582058890161676fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff83fdbab2b9417eac50495c54f0da54. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:09.144346 10928 consensus_queue.cc:1035] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:09.144870 10928 consensus_queue.cc:1035] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:09.144145 10988 tablet_copy_client.cc:806] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:09.156021 10928 raft_consensus.cc:2953] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:09.156662 7932 raft_consensus.cc:2953] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:09.160957 10988 tablet_copy_client.cc:670] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:09.164227 7658 catalog_manager.cc:5582] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index -1 to 4, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:09.167285 10975 tablet_bootstrap.cc:492] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157: Bootstrap starting.
I20250627 13:54:09.167325 8191 raft_consensus.cc:2953] T ff83fdbab2b9417eac50495c54f0da54 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:09.170028 10961 tablet_copy_client.cc:670] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:09.173897 10961 tablet_copy_client.cc:538] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:09.176477 10953 ts_tablet_manager.cc:927] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a: Initiating tablet copy from peer 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:54:09.181011 10988 tablet_copy_client.cc:538] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:09.186836 10975 log.cc:826] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157: Log is configured to *not* fsync() on all Append() calls
I20250627 13:54:09.196559 10961 tablet_bootstrap.cc:492] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap starting.
I20250627 13:54:09.207733 7800 raft_consensus.cc:1273] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:09.208321 10632 consensus_queue.cc:1035] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:09.220616 8191 raft_consensus.cc:1273] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:09.221213 10632 consensus_queue.cc:1035] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:09.232367 10988 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157: Bootstrap starting.
I20250627 13:54:09.234620 10953 tablet_copy_client.cc:323] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:37467
I20250627 13:54:09.242880 7822 tablet_copy_service.cc:140] P 83574c1bf8ea45adaff939158f93c096: Received BeginTabletCopySession request for tablet 87726824b05a4b968386f57d3a7a4aad from peer 2a1282ebe92a4c9a92eed3b13f69012a ({username='slave'} at 127.0.103.69:43835)
I20250627 13:54:09.243114 7822 tablet_copy_service.cc:161] P 83574c1bf8ea45adaff939158f93c096: Beginning new tablet copy session on tablet 87726824b05a4b968386f57d3a7a4aad from peer 2a1282ebe92a4c9a92eed3b13f69012a at {username='slave'} at 127.0.103.69:43835: session id = 2a1282ebe92a4c9a92eed3b13f69012a-87726824b05a4b968386f57d3a7a4aad
I20250627 13:54:09.247203 7822 tablet_copy_source_session.cc:215] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:09.247931 8191 consensus_queue.cc:237] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:09.249774 10953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87726824b05a4b968386f57d3a7a4aad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:09.270504 7800 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:09.278139 11024 ts_tablet_manager.cc:927] T 570b134177314b31851951c411a9f4bf P 956ff058e6d64579a1963aabdb6d466b: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:09.277896 8193 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:09.259110 11022 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:09.279163 7658 catalog_manager.cc:5582] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index -1 to 4, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:09.288427 10961 log.cc:826] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b: Log is configured to *not* fsync() on all Append() calls
I20250627 13:54:09.289465 11024 tablet_copy_client.cc:323] T 570b134177314b31851951c411a9f4bf P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:09.292191 8083 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet 570b134177314b31851951c411a9f4bf from peer 956ff058e6d64579a1963aabdb6d466b ({username='slave'} at 127.0.103.70:59981)
I20250627 13:54:09.292373 8083 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet 570b134177314b31851951c411a9f4bf from peer 956ff058e6d64579a1963aabdb6d466b at {username='slave'} at 127.0.103.70:59981: session id = 956ff058e6d64579a1963aabdb6d466b-570b134177314b31851951c411a9f4bf
I20250627 13:54:09.295409 8083 tablet_copy_source_session.cc:215] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:09.297847 11024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 570b134177314b31851951c411a9f4bf. 1 dirs total, 0 dirs full, 0 dirs failed
W20250627 13:54:09.306927 7869 consensus_peers.cc:487] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 -> Peer 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101): Couldn't send request to peer 2a1282ebe92a4c9a92eed3b13f69012a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b265cfda4bfe48908659208ec8eeab2d. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:09.309186 7931 raft_consensus.cc:1273] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:09.309230 10953 tablet_copy_client.cc:806] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:09.310106 10583 consensus_queue.cc:1035] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:09.311034 10953 tablet_copy_client.cc:670] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:09.313400 10953 tablet_copy_client.cc:538] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:09.316463 7800 raft_consensus.cc:1273] T da90822ff15e4e5da05114ef38d3970a P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:09.317555 10583 consensus_queue.cc:1035] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:09.334748 10953 tablet_bootstrap.cc:492] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap starting.
I20250627 13:54:09.340824 11024 tablet_copy_client.cc:806] T 570b134177314b31851951c411a9f4bf P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:09.341135 11024 tablet_copy_client.cc:670] T 570b134177314b31851951c411a9f4bf P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:09.344580 11024 tablet_copy_client.cc:538] T 570b134177314b31851951c411a9f4bf P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:09.348922 11024 tablet_bootstrap.cc:492] T 570b134177314b31851951c411a9f4bf P 956ff058e6d64579a1963aabdb6d466b: Bootstrap starting.
I20250627 13:54:09.354156 11033 ts_tablet_manager.cc:927] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:09.357621 10583 raft_consensus.cc:2953] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:09.367262 7801 raft_consensus.cc:2953] T da90822ff15e4e5da05114ef38d3970a P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:09.372953 7658 catalog_manager.cc:5582] T da90822ff15e4e5da05114ef38d3970a P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:09.399716 11033 tablet_copy_client.cc:323] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:09.401355 11044 ts_tablet_manager.cc:927] T 720cd645f0b7414f819fcbff3fd262c3 P 6686c6c365bf4f4582058890161676fe: Initiating tablet copy from peer 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:54:09.409260 9711 raft_consensus.cc:1215] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250627 13:54:09.414487 11044 tablet_copy_client.cc:323] T 720cd645f0b7414f819fcbff3fd262c3 P 6686c6c365bf4f4582058890161676fe: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:37467
I20250627 13:54:09.415622 8083 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet 91664403eed74badbeba83d69f597e1f from peer dede81a8c0164427a61532d01b6fc157 ({username='slave'} at 127.0.103.72:50607)
I20250627 13:54:09.415797 8083 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet 91664403eed74badbeba83d69f597e1f from peer dede81a8c0164427a61532d01b6fc157 at {username='slave'} at 127.0.103.72:50607: session id = dede81a8c0164427a61532d01b6fc157-91664403eed74badbeba83d69f597e1f
W20250627 13:54:09.415824 8126 consensus_peers.cc:487] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da90822ff15e4e5da05114ef38d3970a. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:09.419611 8083 tablet_copy_source_session.cc:215] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:09.419409 7931 raft_consensus.cc:2953] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:09.421429 11033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91664403eed74badbeba83d69f597e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:09.439087 8191 consensus_queue.cc:237] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:09.455510 11056 ts_tablet_manager.cc:927] T c322cfebf12c4fef8d88fbd68e7e8b6f P 2a1282ebe92a4c9a92eed3b13f69012a: Initiating tablet copy from peer 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:54:09.462543 11033 tablet_copy_client.cc:806] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:09.462808 11033 tablet_copy_client.cc:670] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:09.467454 10975 tablet_bootstrap.cc:492] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:09.468152 10975 tablet_bootstrap.cc:492] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157: Bootstrap complete.
I20250627 13:54:09.468667 11056 tablet_copy_client.cc:323] T c322cfebf12c4fef8d88fbd68e7e8b6f P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:37467
I20250627 13:54:09.468772 10975 ts_tablet_manager.cc:1397] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157: Time spent bootstrapping tablet: real 0.302s user 0.056s sys 0.018s
I20250627 13:54:09.469542 7822 tablet_copy_service.cc:140] P 83574c1bf8ea45adaff939158f93c096: Received BeginTabletCopySession request for tablet c322cfebf12c4fef8d88fbd68e7e8b6f from peer 2a1282ebe92a4c9a92eed3b13f69012a ({username='slave'} at 127.0.103.69:43835)
I20250627 13:54:09.469779 7822 tablet_copy_service.cc:161] P 83574c1bf8ea45adaff939158f93c096: Beginning new tablet copy session on tablet c322cfebf12c4fef8d88fbd68e7e8b6f from peer 2a1282ebe92a4c9a92eed3b13f69012a at {username='slave'} at 127.0.103.69:43835: session id = 2a1282ebe92a4c9a92eed3b13f69012a-c322cfebf12c4fef8d88fbd68e7e8b6f
I20250627 13:54:09.472868 7822 tablet_copy_source_session.cc:215] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:09.474223 11056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c322cfebf12c4fef8d88fbd68e7e8b6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:09.480340 11033 tablet_copy_client.cc:538] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:09.482689 7931 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:09.483326 11053 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:09.484103 11033 tablet_bootstrap.cc:492] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157: Bootstrap starting.
I20250627 13:54:09.490108 7800 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:09.491220 11053 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:09.492507 11056 tablet_copy_client.cc:806] T c322cfebf12c4fef8d88fbd68e7e8b6f P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:09.482864 10975 raft_consensus.cc:357] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:09.494372 10975 raft_consensus.cc:738] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dede81a8c0164427a61532d01b6fc157, State: Initialized, Role: LEARNER
I20250627 13:54:09.494762 11056 tablet_copy_client.cc:670] T c322cfebf12c4fef8d88fbd68e7e8b6f P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:09.496579 10975 consensus_queue.cc:260] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:09.497125 11056 tablet_copy_client.cc:538] T c322cfebf12c4fef8d88fbd68e7e8b6f P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:09.501173 10975 ts_tablet_manager.cc:1428] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157: Time spent starting tablet: real 0.032s user 0.009s sys 0.001s
I20250627 13:54:09.510157 11056 tablet_bootstrap.cc:492] T c322cfebf12c4fef8d88fbd68e7e8b6f P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap starting.
I20250627 13:54:09.510411 11053 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:09.513232 8062 consensus_queue.cc:237] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:09.521018 10104 raft_consensus.cc:1215] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20250627 13:54:09.526913 7929 raft_consensus.cc:1273] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20250627 13:54:09.527042 7998 consensus_peers.cc:487] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213): Couldn't send request to peer dede81a8c0164427a61532d01b6fc157. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96fe0309adc54fb6a426bae02f26f898. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:09.527704 8193 raft_consensus.cc:1273] T 96fe0309adc54fb6a426bae02f26f898 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:09.527952 7656 catalog_manager.cc:5582] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:09.528223 8083 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session dede81a8c0164427a61532d01b6fc157-d2eb537d49804e1b8ec5a217b1564988 received from {username='slave'} at 127.0.103.72:50607
I20250627 13:54:09.528388 8083 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session dede81a8c0164427a61532d01b6fc157-d2eb537d49804e1b8ec5a217b1564988 on tablet d2eb537d49804e1b8ec5a217b1564988 with peer dede81a8c0164427a61532d01b6fc157
I20250627 13:54:09.528437 10933 consensus_queue.cc:1035] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:09.530611 10933 consensus_queue.cc:1035] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:09.538890 7800 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:09.545008 7931 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:09.560745 10933 raft_consensus.cc:2953] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:09.567077 8191 raft_consensus.cc:2953] T 96fe0309adc54fb6a426bae02f26f898 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:09.571221 7658 catalog_manager.cc:5582] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:09.573246 7932 consensus_queue.cc:237] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:09.578773 10961 tablet_bootstrap.cc:492] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:09.579473 10961 tablet_bootstrap.cc:492] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap complete.
I20250627 13:54:09.585812 10961 ts_tablet_manager.cc:1397] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b: Time spent bootstrapping tablet: real 0.390s user 0.063s sys 0.012s
I20250627 13:54:09.585799 7822 tablet_copy_service.cc:140] P 83574c1bf8ea45adaff939158f93c096: Received BeginTabletCopySession request for tablet 720cd645f0b7414f819fcbff3fd262c3 from peer 6686c6c365bf4f4582058890161676fe ({username='slave'} at 127.0.103.71:53529)
I20250627 13:54:09.585942 7822 tablet_copy_service.cc:161] P 83574c1bf8ea45adaff939158f93c096: Beginning new tablet copy session on tablet 720cd645f0b7414f819fcbff3fd262c3 from peer 6686c6c365bf4f4582058890161676fe at {username='slave'} at 127.0.103.71:53529: session id = 6686c6c365bf4f4582058890161676fe-720cd645f0b7414f819fcbff3fd262c3
I20250627 13:54:09.590116 7822 tablet_copy_source_session.cc:215] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:09.595522 11044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 720cd645f0b7414f819fcbff3fd262c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:09.596714 10988 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:09.597116 10988 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157: Bootstrap complete.
I20250627 13:54:09.597373 10988 ts_tablet_manager.cc:1397] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157: Time spent bootstrapping tablet: real 0.365s user 0.048s sys 0.010s
I20250627 13:54:09.599704 10961 raft_consensus.cc:357] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
W20250627 13:54:09.600071 8128 consensus_peers.cc:487] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e -> Peer dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213): Couldn't send request to peer dede81a8c0164427a61532d01b6fc157. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14dba72d45694defb83f41c1572722ca. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:09.600157 10961 raft_consensus.cc:738] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 956ff058e6d64579a1963aabdb6d466b, State: Initialized, Role: LEARNER
I20250627 13:54:09.602108 10988 raft_consensus.cc:357] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:09.602350 10988 raft_consensus.cc:738] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dede81a8c0164427a61532d01b6fc157, State: Initialized, Role: LEARNER
I20250627 13:54:09.602533 10988 consensus_queue.cc:260] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:09.603016 10988 ts_tablet_manager.cc:1428] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250627 13:54:09.603819 8083 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session dede81a8c0164427a61532d01b6fc157-644985e3086d4614b6598a727cb60c1d received from {username='slave'} at 127.0.103.72:50607
I20250627 13:54:09.603956 8083 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session dede81a8c0164427a61532d01b6fc157-644985e3086d4614b6598a727cb60c1d on tablet 644985e3086d4614b6598a727cb60c1d with peer dede81a8c0164427a61532d01b6fc157
I20250627 13:54:09.604105 10961 consensus_queue.cc:260] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:09.620759 7929 raft_consensus.cc:2953] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:09.622051 10961 ts_tablet_manager.cc:1428] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b: Time spent starting tablet: real 0.036s user 0.009s sys 0.003s
I20250627 13:54:09.622902 8083 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session 956ff058e6d64579a1963aabdb6d466b-d2eb537d49804e1b8ec5a217b1564988 received from {username='slave'} at 127.0.103.70:59981
I20250627 13:54:09.623095 8083 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session 956ff058e6d64579a1963aabdb6d466b-d2eb537d49804e1b8ec5a217b1564988 on tablet d2eb537d49804e1b8ec5a217b1564988 with peer 956ff058e6d64579a1963aabdb6d466b
I20250627 13:54:09.632068 8062 consensus_queue.cc:237] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:09.636682 8061 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:09.637323 11021 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:09.642328 8191 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20250627 13:54:09.642921 7866 consensus_peers.cc:487] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03640a5124d2405faf4a26d0e22e1ffc. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:09.643641 11021 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:09.650707 11021 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:09.651034 8061 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:09.653141 7932 raft_consensus.cc:1273] T eed3236c4fe8455cb9527b9f4b754e81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:09.653698 10933 consensus_queue.cc:1035] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:09.653846 7800 raft_consensus.cc:1273] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:09.654517 10933 consensus_queue.cc:1035] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:09.663072 7656 catalog_manager.cc:5582] T 03640a5124d2405faf4a26d0e22e1ffc P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:09.666837 11033 tablet_bootstrap.cc:492] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:09.667347 11033 tablet_bootstrap.cc:492] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157: Bootstrap complete.
I20250627 13:54:09.667675 11033 ts_tablet_manager.cc:1397] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157: Time spent bootstrapping tablet: real 0.184s user 0.054s sys 0.000s
I20250627 13:54:09.668625 11033 raft_consensus.cc:357] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:09.668906 11033 raft_consensus.cc:738] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dede81a8c0164427a61532d01b6fc157, State: Initialized, Role: LEARNER
I20250627 13:54:09.678786 11033 consensus_queue.cc:260] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:09.680439 11033 ts_tablet_manager.cc:1428] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20250627 13:54:09.681241 10933 raft_consensus.cc:2953] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:09.686523 7801 raft_consensus.cc:2953] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:09.687747 8083 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session dede81a8c0164427a61532d01b6fc157-91664403eed74badbeba83d69f597e1f received from {username='slave'} at 127.0.103.72:50607
I20250627 13:54:09.687906 8083 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session dede81a8c0164427a61532d01b6fc157-91664403eed74badbeba83d69f597e1f on tablet 91664403eed74badbeba83d69f597e1f with peer dede81a8c0164427a61532d01b6fc157
I20250627 13:54:09.688988 10961 ts_tablet_manager.cc:927] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b: Initiating tablet copy from peer e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:54:09.693622 7658 catalog_manager.cc:5582] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index -1 to 4, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
W20250627 13:54:09.700207 7999 consensus_peers.cc:487] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101): Couldn't send request to peer 2a1282ebe92a4c9a92eed3b13f69012a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eed3236c4fe8455cb9527b9f4b754e81. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:09.716850 11098 ts_tablet_manager.cc:927] T ff83fdbab2b9417eac50495c54f0da54 P 6686c6c365bf4f4582058890161676fe: Initiating tablet copy from peer 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:54:09.722056 8192 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:09.722819 10961 tablet_copy_client.cc:323] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.68:46139
I20250627 13:54:09.724891 7801 consensus_queue.cc:237] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:09.736810 7931 raft_consensus.cc:2953] T eed3236c4fe8455cb9527b9f4b754e81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:09.739687 11044 tablet_copy_client.cc:806] T 720cd645f0b7414f819fcbff3fd262c3 P 6686c6c365bf4f4582058890161676fe: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:09.742959 11044 tablet_copy_client.cc:670] T 720cd645f0b7414f819fcbff3fd262c3 P 6686c6c365bf4f4582058890161676fe: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:09.745404 11044 tablet_copy_client.cc:538] T 720cd645f0b7414f819fcbff3fd262c3 P 6686c6c365bf4f4582058890161676fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:09.748402 11098 tablet_copy_client.cc:323] T ff83fdbab2b9417eac50495c54f0da54 P 6686c6c365bf4f4582058890161676fe: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:37467
I20250627 13:54:09.749194 7822 tablet_copy_service.cc:140] P 83574c1bf8ea45adaff939158f93c096: Received BeginTabletCopySession request for tablet ff83fdbab2b9417eac50495c54f0da54 from peer 6686c6c365bf4f4582058890161676fe ({username='slave'} at 127.0.103.71:53529)
I20250627 13:54:09.749408 7822 tablet_copy_service.cc:161] P 83574c1bf8ea45adaff939158f93c096: Beginning new tablet copy session on tablet ff83fdbab2b9417eac50495c54f0da54 from peer 6686c6c365bf4f4582058890161676fe at {username='slave'} at 127.0.103.71:53529: session id = 6686c6c365bf4f4582058890161676fe-ff83fdbab2b9417eac50495c54f0da54
I20250627 13:54:09.752781 7822 tablet_copy_source_session.cc:215] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:09.754298 11098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff83fdbab2b9417eac50495c54f0da54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:09.767717 11044 tablet_bootstrap.cc:492] T 720cd645f0b7414f819fcbff3fd262c3 P 6686c6c365bf4f4582058890161676fe: Bootstrap starting.
I20250627 13:54:09.770071 11098 tablet_copy_client.cc:806] T ff83fdbab2b9417eac50495c54f0da54 P 6686c6c365bf4f4582058890161676fe: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:09.770295 11098 tablet_copy_client.cc:670] T ff83fdbab2b9417eac50495c54f0da54 P 6686c6c365bf4f4582058890161676fe: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:09.771016 11110 ts_tablet_manager.cc:927] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a: Initiating tablet copy from peer a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
W20250627 13:54:09.774593 7738 consensus_peers.cc:487] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 -> Peer 6686c6c365bf4f4582058890161676fe (127.0.103.71:37859): Couldn't send request to peer 6686c6c365bf4f4582058890161676fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40431e3bd7244904a4e65f59477bd21d. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:09.775089 7929 raft_consensus.cc:1273] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:09.775653 10926 consensus_queue.cc:1035] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:09.783274 10953 tablet_bootstrap.cc:492] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:09.785342 10953 tablet_bootstrap.cc:492] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap complete.
I20250627 13:54:09.786329 10953 ts_tablet_manager.cc:1397] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a: Time spent bootstrapping tablet: real 0.452s user 0.055s sys 0.001s
I20250627 13:54:09.787523 10953 raft_consensus.cc:357] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:09.789013 11098 tablet_copy_client.cc:538] T ff83fdbab2b9417eac50495c54f0da54 P 6686c6c365bf4f4582058890161676fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:09.788996 8192 raft_consensus.cc:1273] T 40431e3bd7244904a4e65f59477bd21d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:09.789618 10926 consensus_queue.cc:1035] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:09.792626 11098 tablet_bootstrap.cc:492] T ff83fdbab2b9417eac50495c54f0da54 P 6686c6c365bf4f4582058890161676fe: Bootstrap starting.
I20250627 13:54:09.797065 11044 log.cc:826] T 720cd645f0b7414f819fcbff3fd262c3 P 6686c6c365bf4f4582058890161676fe: Log is configured to *not* fsync() on all Append() calls
I20250627 13:54:09.797580 10953 raft_consensus.cc:738] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a1282ebe92a4c9a92eed3b13f69012a, State: Initialized, Role: LEARNER
I20250627 13:54:09.800205 10953 consensus_queue.cc:260] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:09.801888 10953 ts_tablet_manager.cc:1428] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20250627 13:54:09.801903 10926 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:09.806263 8191 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:09.807049 7822 tablet_copy_service.cc:342] P 83574c1bf8ea45adaff939158f93c096: Request end of tablet copy session 2a1282ebe92a4c9a92eed3b13f69012a-87726824b05a4b968386f57d3a7a4aad received from {username='slave'} at 127.0.103.69:43835
I20250627 13:54:09.807196 7822 tablet_copy_service.cc:434] P 83574c1bf8ea45adaff939158f93c096: ending tablet copy session 2a1282ebe92a4c9a92eed3b13f69012a-87726824b05a4b968386f57d3a7a4aad on tablet 87726824b05a4b968386f57d3a7a4aad with peer 2a1282ebe92a4c9a92eed3b13f69012a
I20250627 13:54:09.813405 7658 catalog_manager.cc:5582] T 40431e3bd7244904a4e65f59477bd21d P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index -1 to 4, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:09.815557 7929 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:09.846205 11110 tablet_copy_client.cc:323] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:43675
I20250627 13:54:09.851491 9842 raft_consensus.cc:1215] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20250627 13:54:09.855250 8213 tablet_copy_service.cc:140] P e4c831df4da841929d13d6c783099b0e: Received BeginTabletCopySession request for tablet da90822ff15e4e5da05114ef38d3970a from peer 956ff058e6d64579a1963aabdb6d466b ({username='slave'} at 127.0.103.70:33381)
I20250627 13:54:09.855436 8213 tablet_copy_service.cc:161] P e4c831df4da841929d13d6c783099b0e: Beginning new tablet copy session on tablet da90822ff15e4e5da05114ef38d3970a from peer 956ff058e6d64579a1963aabdb6d466b at {username='slave'} at 127.0.103.70:33381: session id = 956ff058e6d64579a1963aabdb6d466b-da90822ff15e4e5da05114ef38d3970a
I20250627 13:54:09.860384 8213 tablet_copy_source_session.cc:215] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:09.861807 10961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da90822ff15e4e5da05114ef38d3970a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:09.863675 8192 consensus_queue.cc:237] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:09.870481 10961 tablet_copy_client.cc:806] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:09.870774 10961 tablet_copy_client.cc:670] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:09.872355 10961 tablet_copy_client.cc:538] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:09.882061 10104 raft_consensus.cc:1215] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250627 13:54:09.883062 10961 tablet_bootstrap.cc:492] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b: Bootstrap starting.
I20250627 13:54:09.883491 11056 tablet_bootstrap.cc:492] T c322cfebf12c4fef8d88fbd68e7e8b6f P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:09.883916 11056 tablet_bootstrap.cc:492] T c322cfebf12c4fef8d88fbd68e7e8b6f P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap complete.
I20250627 13:54:09.884163 11056 ts_tablet_manager.cc:1397] T c322cfebf12c4fef8d88fbd68e7e8b6f P 2a1282ebe92a4c9a92eed3b13f69012a: Time spent bootstrapping tablet: real 0.383s user 0.052s sys 0.002s
I20250627 13:54:09.884935 11056 raft_consensus.cc:357] T c322cfebf12c4fef8d88fbd68e7e8b6f P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:09.885146 11056 raft_consensus.cc:738] T c322cfebf12c4fef8d88fbd68e7e8b6f P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a1282ebe92a4c9a92eed3b13f69012a, State: Initialized, Role: LEARNER
I20250627 13:54:09.885356 11056 consensus_queue.cc:260] T c322cfebf12c4fef8d88fbd68e7e8b6f P 2a1282ebe92a4c9a92eed3b13f69012a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
W20250627 13:54:09.891780 8128 consensus_peers.cc:487] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e -> Peer dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213): Couldn't send request to peer dede81a8c0164427a61532d01b6fc157. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f5e370b77cc42fdb0b886ea19af06c3. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:09.891744 7953 tablet_copy_service.cc:140] P a41a89ae8ed743edb2b5cf4490d8c854: Received BeginTabletCopySession request for tablet b265cfda4bfe48908659208ec8eeab2d from peer 2a1282ebe92a4c9a92eed3b13f69012a ({username='slave'} at 127.0.103.69:49083)
I20250627 13:54:09.891953 7953 tablet_copy_service.cc:161] P a41a89ae8ed743edb2b5cf4490d8c854: Beginning new tablet copy session on tablet b265cfda4bfe48908659208ec8eeab2d from peer 2a1282ebe92a4c9a92eed3b13f69012a at {username='slave'} at 127.0.103.69:49083: session id = 2a1282ebe92a4c9a92eed3b13f69012a-b265cfda4bfe48908659208ec8eeab2d
I20250627 13:54:09.892683 8061 raft_consensus.cc:1273] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:09.893250 7929 raft_consensus.cc:1273] T 6f5e370b77cc42fdb0b886ea19af06c3 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:09.893997 11048 consensus_queue.cc:1035] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:09.895989 7953 tablet_copy_source_session.cc:215] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:09.896025 11048 consensus_queue.cc:1035] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:09.899085 11053 raft_consensus.cc:2953] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:09.899693 7932 raft_consensus.cc:2953] T 6f5e370b77cc42fdb0b886ea19af06c3 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:09.901726 7929 consensus_queue.cc:237] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:09.904464 11033 ts_tablet_manager.cc:927] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157: Initiating tablet copy from peer e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:54:09.911044 11056 ts_tablet_manager.cc:1428] T c322cfebf12c4fef8d88fbd68e7e8b6f P 2a1282ebe92a4c9a92eed3b13f69012a: Time spent starting tablet: real 0.026s user 0.003s sys 0.000s
I20250627 13:54:09.911420 7656 catalog_manager.cc:5582] T 6f5e370b77cc42fdb0b886ea19af06c3 P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:09.911944 7822 tablet_copy_service.cc:342] P 83574c1bf8ea45adaff939158f93c096: Request end of tablet copy session 2a1282ebe92a4c9a92eed3b13f69012a-c322cfebf12c4fef8d88fbd68e7e8b6f received from {username='slave'} at 127.0.103.69:43835
I20250627 13:54:09.912070 7822 tablet_copy_service.cc:434] P 83574c1bf8ea45adaff939158f93c096: ending tablet copy session 2a1282ebe92a4c9a92eed3b13f69012a-c322cfebf12c4fef8d88fbd68e7e8b6f on tablet c322cfebf12c4fef8d88fbd68e7e8b6f with peer 2a1282ebe92a4c9a92eed3b13f69012a
W20250627 13:54:09.913239 7869 consensus_peers.cc:487] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 -> Peer 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101): Couldn't send request to peer 2a1282ebe92a4c9a92eed3b13f69012a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03640a5124d2405faf4a26d0e22e1ffc. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:09.914614 8192 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:09.915154 11021 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:09.919006 11021 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:09.922341 8192 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:09.923024 11110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b265cfda4bfe48908659208ec8eeab2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:09.930272 7653 catalog_manager.cc:5582] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 4 to 5, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:09.942334 11033 tablet_copy_client.cc:323] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.68:46139
I20250627 13:54:09.958046 11110 tablet_copy_client.cc:806] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:09.958351 11110 tablet_copy_client.cc:670] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:09.959098 11098 tablet_bootstrap.cc:492] T ff83fdbab2b9417eac50495c54f0da54 P 6686c6c365bf4f4582058890161676fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:09.962319 11098 tablet_bootstrap.cc:492] T ff83fdbab2b9417eac50495c54f0da54 P 6686c6c365bf4f4582058890161676fe: Bootstrap complete.
I20250627 13:54:09.962841 11098 ts_tablet_manager.cc:1397] T ff83fdbab2b9417eac50495c54f0da54 P 6686c6c365bf4f4582058890161676fe: Time spent bootstrapping tablet: real 0.170s user 0.047s sys 0.010s
I20250627 13:54:09.963471 8213 tablet_copy_service.cc:140] P e4c831df4da841929d13d6c783099b0e: Received BeginTabletCopySession request for tablet 14dba72d45694defb83f41c1572722ca from peer dede81a8c0164427a61532d01b6fc157 ({username='slave'} at 127.0.103.72:33995)
I20250627 13:54:09.963641 8213 tablet_copy_service.cc:161] P e4c831df4da841929d13d6c783099b0e: Beginning new tablet copy session on tablet 14dba72d45694defb83f41c1572722ca from peer dede81a8c0164427a61532d01b6fc157 at {username='slave'} at 127.0.103.72:33995: session id = dede81a8c0164427a61532d01b6fc157-14dba72d45694defb83f41c1572722ca
I20250627 13:54:09.966884 8213 tablet_copy_source_session.cc:215] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:09.968026 11033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14dba72d45694defb83f41c1572722ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:09.968385 11098 raft_consensus.cc:357] T ff83fdbab2b9417eac50495c54f0da54 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:09.968920 11098 raft_consensus.cc:738] T ff83fdbab2b9417eac50495c54f0da54 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6686c6c365bf4f4582058890161676fe, State: Initialized, Role: LEARNER
I20250627 13:54:09.969565 11098 consensus_queue.cc:260] T ff83fdbab2b9417eac50495c54f0da54 P 6686c6c365bf4f4582058890161676fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:09.977245 11033 tablet_copy_client.cc:806] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:09.977500 11033 tablet_copy_client.cc:670] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:09.977789 11110 tablet_copy_client.cc:538] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:09.979486 11048 raft_consensus.cc:1062] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: attempting to promote NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a to VOTER
I20250627 13:54:09.980484 11048 consensus_queue.cc:237] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:09.981237 11110 tablet_bootstrap.cc:492] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap starting.
I20250627 13:54:09.982710 11033 tablet_copy_client.cc:538] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:09.987136 11033 tablet_bootstrap.cc:492] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157: Bootstrap starting.
I20250627 13:54:09.991883 11044 tablet_bootstrap.cc:492] T 720cd645f0b7414f819fcbff3fd262c3 P 6686c6c365bf4f4582058890161676fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:09.992455 11044 tablet_bootstrap.cc:492] T 720cd645f0b7414f819fcbff3fd262c3 P 6686c6c365bf4f4582058890161676fe: Bootstrap complete.
I20250627 13:54:09.992775 11044 ts_tablet_manager.cc:1397] T 720cd645f0b7414f819fcbff3fd262c3 P 6686c6c365bf4f4582058890161676fe: Time spent bootstrapping tablet: real 0.225s user 0.065s sys 0.006s
W20250627 13:54:09.993904 7866 consensus_peers.cc:487] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03640a5124d2405faf4a26d0e22e1ffc. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:09.994025 11044 raft_consensus.cc:357] T 720cd645f0b7414f819fcbff3fd262c3 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:09.994254 11044 raft_consensus.cc:738] T 720cd645f0b7414f819fcbff3fd262c3 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6686c6c365bf4f4582058890161676fe, State: Initialized, Role: LEARNER
I20250627 13:54:09.994488 11044 consensus_queue.cc:260] T 720cd645f0b7414f819fcbff3fd262c3 P 6686c6c365bf4f4582058890161676fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:10.002534 11044 ts_tablet_manager.cc:1428] T 720cd645f0b7414f819fcbff3fd262c3 P 6686c6c365bf4f4582058890161676fe: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20250627 13:54:10.003578 7822 tablet_copy_service.cc:342] P 83574c1bf8ea45adaff939158f93c096: Request end of tablet copy session 6686c6c365bf4f4582058890161676fe-720cd645f0b7414f819fcbff3fd262c3 received from {username='slave'} at 127.0.103.71:53529
I20250627 13:54:10.003733 7822 tablet_copy_service.cc:434] P 83574c1bf8ea45adaff939158f93c096: ending tablet copy session 6686c6c365bf4f4582058890161676fe-720cd645f0b7414f819fcbff3fd262c3 on tablet 720cd645f0b7414f819fcbff3fd262c3 with peer 6686c6c365bf4f4582058890161676fe
I20250627 13:54:10.008843 11098 ts_tablet_manager.cc:1428] T ff83fdbab2b9417eac50495c54f0da54 P 6686c6c365bf4f4582058890161676fe: Time spent starting tablet: real 0.046s user 0.013s sys 0.022s
I20250627 13:54:10.012511 8060 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.013242 7822 tablet_copy_service.cc:342] P 83574c1bf8ea45adaff939158f93c096: Request end of tablet copy session 6686c6c365bf4f4582058890161676fe-ff83fdbab2b9417eac50495c54f0da54 received from {username='slave'} at 127.0.103.71:53529
I20250627 13:54:10.013355 7822 tablet_copy_service.cc:434] P 83574c1bf8ea45adaff939158f93c096: ending tablet copy session 6686c6c365bf4f4582058890161676fe-ff83fdbab2b9417eac50495c54f0da54 on tablet ff83fdbab2b9417eac50495c54f0da54 with peer 6686c6c365bf4f4582058890161676fe
I20250627 13:54:10.014547 11021 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:10.015421 10988 ts_tablet_manager.cc:927] T 96fe0309adc54fb6a426bae02f26f898 P dede81a8c0164427a61532d01b6fc157: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:10.026010 8060 raft_consensus.cc:2953] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:10.028512 9711 raft_consensus.cc:1215] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250627 13:54:10.044180 8060 consensus_queue.cc:237] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:10.046077 8062 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.046888 7801 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.047842 9711 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.048496 11034 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:10.049093 11034 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:10.052122 11024 tablet_bootstrap.cc:492] T 570b134177314b31851951c411a9f4bf P 956ff058e6d64579a1963aabdb6d466b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:10.052603 11024 tablet_bootstrap.cc:492] T 570b134177314b31851951c411a9f4bf P 956ff058e6d64579a1963aabdb6d466b: Bootstrap complete.
I20250627 13:54:10.052902 11024 ts_tablet_manager.cc:1397] T 570b134177314b31851951c411a9f4bf P 956ff058e6d64579a1963aabdb6d466b: Time spent bootstrapping tablet: real 0.704s user 0.058s sys 0.004s
I20250627 13:54:10.053143 11034 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:10.053893 11024 raft_consensus.cc:357] T 570b134177314b31851951c411a9f4bf P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:10.054109 11024 raft_consensus.cc:738] T 570b134177314b31851951c411a9f4bf P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 956ff058e6d64579a1963aabdb6d466b, State: Initialized, Role: LEARNER
I20250627 13:54:10.054303 11024 consensus_queue.cc:260] T 570b134177314b31851951c411a9f4bf P 956ff058e6d64579a1963aabdb6d466b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:10.056572 10988 tablet_copy_client.cc:323] T 96fe0309adc54fb6a426bae02f26f898 P dede81a8c0164427a61532d01b6fc157: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:10.057186 8083 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet 96fe0309adc54fb6a426bae02f26f898 from peer dede81a8c0164427a61532d01b6fc157 ({username='slave'} at 127.0.103.72:50607)
I20250627 13:54:10.057356 8083 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet 96fe0309adc54fb6a426bae02f26f898 from peer dede81a8c0164427a61532d01b6fc157 at {username='slave'} at 127.0.103.72:50607: session id = dede81a8c0164427a61532d01b6fc157-96fe0309adc54fb6a426bae02f26f898
I20250627 13:54:10.064839 11033 tablet_bootstrap.cc:492] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:10.064867 11024 ts_tablet_manager.cc:1428] T 570b134177314b31851951c411a9f4bf P 956ff058e6d64579a1963aabdb6d466b: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250627 13:54:10.065380 11033 tablet_bootstrap.cc:492] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157: Bootstrap complete.
I20250627 13:54:10.065696 11033 ts_tablet_manager.cc:1397] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157: Time spent bootstrapping tablet: real 0.079s user 0.059s sys 0.005s
I20250627 13:54:10.066601 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session 956ff058e6d64579a1963aabdb6d466b-570b134177314b31851951c411a9f4bf received from {username='slave'} at 127.0.103.70:59981
I20250627 13:54:10.066758 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session 956ff058e6d64579a1963aabdb6d466b-570b134177314b31851951c411a9f4bf on tablet 570b134177314b31851951c411a9f4bf with peer 956ff058e6d64579a1963aabdb6d466b
I20250627 13:54:10.067261 11033 raft_consensus.cc:357] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:10.067482 11033 raft_consensus.cc:738] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dede81a8c0164427a61532d01b6fc157, State: Initialized, Role: LEARNER
I20250627 13:54:10.067723 11033 consensus_queue.cc:260] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:10.068950 11033 ts_tablet_manager.cc:1428] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250627 13:54:10.069701 8213 tablet_copy_service.cc:342] P e4c831df4da841929d13d6c783099b0e: Request end of tablet copy session dede81a8c0164427a61532d01b6fc157-14dba72d45694defb83f41c1572722ca received from {username='slave'} at 127.0.103.72:33995
I20250627 13:54:10.069875 8213 tablet_copy_service.cc:434] P e4c831df4da841929d13d6c783099b0e: ending tablet copy session dede81a8c0164427a61532d01b6fc157-14dba72d45694defb83f41c1572722ca on tablet 14dba72d45694defb83f41c1572722ca with peer dede81a8c0164427a61532d01b6fc157
I20250627 13:54:10.075937 11056 ts_tablet_manager.cc:927] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:10.086036 8083 tablet_copy_source_session.cc:215] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:10.088635 10988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96fe0309adc54fb6a426bae02f26f898. 1 dirs total, 0 dirs full, 0 dirs failed
W20250627 13:54:10.089165 7998 consensus_peers.cc:487] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213): Couldn't send request to peer dede81a8c0164427a61532d01b6fc157. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96fe0309adc54fb6a426bae02f26f898. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:10.089335 8192 raft_consensus.cc:1273] T 96fe0309adc54fb6a426bae02f26f898 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.090121 10713 consensus_queue.cc:1035] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:10.090209 7929 raft_consensus.cc:1273] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.090806 10713 consensus_queue.cc:1035] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:10.092621 10713 raft_consensus.cc:2953] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:10.094018 7929 raft_consensus.cc:2953] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:10.095012 8192 raft_consensus.cc:2953] T 96fe0309adc54fb6a426bae02f26f898 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:10.101686 7656 catalog_manager.cc:5582] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index 4 to 5, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
W20250627 13:54:10.108269 7999 consensus_peers.cc:487] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 6686c6c365bf4f4582058890161676fe (127.0.103.71:37859): Couldn't send request to peer 6686c6c365bf4f4582058890161676fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96fe0309adc54fb6a426bae02f26f898. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:10.136189 10104 raft_consensus.cc:1215] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250627 13:54:10.140940 10874 raft_consensus.cc:1062] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER dede81a8c0164427a61532d01b6fc157 to VOTER
I20250627 13:54:10.142234 10874 consensus_queue.cc:237] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:10.148139 10988 tablet_copy_client.cc:806] T 96fe0309adc54fb6a426bae02f26f898 P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:10.148358 10988 tablet_copy_client.cc:670] T 96fe0309adc54fb6a426bae02f26f898 P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:10.146498 8060 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:10.163678 10104 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:10.163873 7800 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:10.164568 10933 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:10.164858 9842 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:10.165016 10933 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:10.166810 10933 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:10.174614 11056 tablet_copy_client.cc:323] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:10.176028 11051 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:10.179561 7929 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:10.180120 10713 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:10.183735 7656 catalog_manager.cc:5582] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:10.186111 7801 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:10.186955 8062 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:10.195722 10988 tablet_copy_client.cc:538] T 96fe0309adc54fb6a426bae02f26f898 P dede81a8c0164427a61532d01b6fc157: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:10.193517 9974 raft_consensus.cc:1215] T ff83fdbab2b9417eac50495c54f0da54 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250627 13:54:10.199681 10988 tablet_bootstrap.cc:492] T 96fe0309adc54fb6a426bae02f26f898 P dede81a8c0164427a61532d01b6fc157: Bootstrap starting.
I20250627 13:54:10.204922 8083 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet eed3236c4fe8455cb9527b9f4b754e81 from peer 2a1282ebe92a4c9a92eed3b13f69012a ({username='slave'} at 127.0.103.69:59515)
I20250627 13:54:10.205817 8083 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet eed3236c4fe8455cb9527b9f4b754e81 from peer 2a1282ebe92a4c9a92eed3b13f69012a at {username='slave'} at 127.0.103.69:59515: session id = 2a1282ebe92a4c9a92eed3b13f69012a-eed3236c4fe8455cb9527b9f4b754e81
I20250627 13:54:10.208676 8083 tablet_copy_source_session.cc:215] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:10.210187 11056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eed3236c4fe8455cb9527b9f4b754e81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:10.215095 10713 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:10.219679 9842 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:10.220935 10933 raft_consensus.cc:1062] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER 956ff058e6d64579a1963aabdb6d466b to VOTER
I20250627 13:54:10.220347 10104 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:10.221566 10933 consensus_queue.cc:237] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:10.223906 7658 catalog_manager.cc:5582] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 5 to 6, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20250627 13:54:10.226172 9710 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:10.231364 10104 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:10.234824 9842 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:10.235126 11110 tablet_bootstrap.cc:492] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:10.235563 11110 tablet_bootstrap.cc:492] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap complete.
I20250627 13:54:10.235357 11160 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
I20250627 13:54:10.236956 8192 consensus_queue.cc:237] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:10.237396 9974 raft_consensus.cc:1215] T 720cd645f0b7414f819fcbff3fd262c3 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250627 13:54:10.238559 9711 raft_consensus.cc:1215] T c322cfebf12c4fef8d88fbd68e7e8b6f P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250627 13:54:10.239485 11110 ts_tablet_manager.cc:1397] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a: Time spent bootstrapping tablet: real 0.255s user 0.058s sys 0.010s
I20250627 13:54:10.239992 11188 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:10.240270 11110 raft_consensus.cc:357] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:10.240484 11110 raft_consensus.cc:738] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a1282ebe92a4c9a92eed3b13f69012a, State: Initialized, Role: LEARNER
I20250627 13:54:10.240669 11110 consensus_queue.cc:260] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:10.242089 9711 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:10.242561 11051 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:10.242661 11110 ts_tablet_manager.cc:1428] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250627 13:54:10.243350 7953 tablet_copy_service.cc:342] P a41a89ae8ed743edb2b5cf4490d8c854: Request end of tablet copy session 2a1282ebe92a4c9a92eed3b13f69012a-b265cfda4bfe48908659208ec8eeab2d received from {username='slave'} at 127.0.103.69:49083
I20250627 13:54:10.243541 7953 tablet_copy_service.cc:434] P a41a89ae8ed743edb2b5cf4490d8c854: ending tablet copy session 2a1282ebe92a4c9a92eed3b13f69012a-b265cfda4bfe48908659208ec8eeab2d on tablet b265cfda4bfe48908659208ec8eeab2d with peer 2a1282ebe92a4c9a92eed3b13f69012a
I20250627 13:54:10.244580 7932 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:10.245028 10583 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:10.245846 10874 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:10.246858 7932 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:10.249605 11098 ts_tablet_manager.cc:927] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe: Initiating tablet copy from peer 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:54:10.249965 7648 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet e3053b529a264e8697a1df9e17eeea81 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:10.252252 7659 catalog_manager.cc:5582] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 5 to 6, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:10.252445 9710 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:10.259168 8061 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:10.260195 11098 tablet_copy_client.cc:323] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:37467
I20250627 13:54:10.260586 10583 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:10.261173 7822 tablet_copy_service.cc:140] P 83574c1bf8ea45adaff939158f93c096: Received BeginTabletCopySession request for tablet 40431e3bd7244904a4e65f59477bd21d from peer 6686c6c365bf4f4582058890161676fe ({username='slave'} at 127.0.103.71:53529)
I20250627 13:54:10.261447 7822 tablet_copy_service.cc:161] P 83574c1bf8ea45adaff939158f93c096: Beginning new tablet copy session on tablet 40431e3bd7244904a4e65f59477bd21d from peer 6686c6c365bf4f4582058890161676fe at {username='slave'} at 127.0.103.71:53529: session id = 6686c6c365bf4f4582058890161676fe-40431e3bd7244904a4e65f59477bd21d
I20250627 13:54:10.263486 8061 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:10.265581 7822 tablet_copy_source_session.cc:215] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:10.267342 11098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40431e3bd7244904a4e65f59477bd21d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:10.268085 7782 tablet_service.cc:1515] Processing DeleteTablet for tablet e3053b529a264e8697a1df9e17eeea81 with delete_type TABLET_DATA_TOMBSTONED (TS 83574c1bf8ea45adaff939158f93c096 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45826
I20250627 13:54:10.276398 7802 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:10.279316 10874 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:10.280539 10874 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:10.280701 7800 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:10.283298 11098 tablet_copy_client.cc:806] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:10.283592 11098 tablet_copy_client.cc:670] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:10.284873 7932 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:10.285383 10104 raft_consensus.cc:1215] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250627 13:54:10.286135 9842 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:10.287037 11033 ts_tablet_manager.cc:927] T 6f5e370b77cc42fdb0b886ea19af06c3 P dede81a8c0164427a61532d01b6fc157: Initiating tablet copy from peer e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:54:10.295086 11098 tablet_copy_client.cc:538] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:10.302083 9841 raft_consensus.cc:1215] T 570b134177314b31851951c411a9f4bf P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250627 13:54:10.305128 11098 tablet_bootstrap.cc:492] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe: Bootstrap starting.
I20250627 13:54:10.306363 11194 tablet_replica.cc:331] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096: stopping tablet replica
I20250627 13:54:10.307025 11194 raft_consensus.cc:2241] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:10.307368 11033 tablet_copy_client.cc:323] T 6f5e370b77cc42fdb0b886ea19af06c3 P dede81a8c0164427a61532d01b6fc157: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.68:46139
I20250627 13:54:10.307428 11194 raft_consensus.cc:2270] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:10.307897 9711 raft_consensus.cc:1215] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250627 13:54:10.308018 8213 tablet_copy_service.cc:140] P e4c831df4da841929d13d6c783099b0e: Received BeginTabletCopySession request for tablet 6f5e370b77cc42fdb0b886ea19af06c3 from peer dede81a8c0164427a61532d01b6fc157 ({username='slave'} at 127.0.103.72:33995)
I20250627 13:54:10.308178 8213 tablet_copy_service.cc:161] P e4c831df4da841929d13d6c783099b0e: Beginning new tablet copy session on tablet 6f5e370b77cc42fdb0b886ea19af06c3 from peer dede81a8c0164427a61532d01b6fc157 at {username='slave'} at 127.0.103.72:33995: session id = dede81a8c0164427a61532d01b6fc157-6f5e370b77cc42fdb0b886ea19af06c3
I20250627 13:54:10.309127 11056 tablet_copy_client.cc:806] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:10.309360 11056 tablet_copy_client.cc:670] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:10.311189 8213 tablet_copy_source_session.cc:215] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:10.311832 11056 tablet_copy_client.cc:538] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:10.314818 11033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f5e370b77cc42fdb0b886ea19af06c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:10.316812 7659 catalog_manager.cc:5582] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b reported cstate change: config changed from index 6 to 7, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:10.323199 10104 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:10.324855 11056 tablet_bootstrap.cc:492] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap starting.
I20250627 13:54:10.335894 11033 tablet_copy_client.cc:806] T 6f5e370b77cc42fdb0b886ea19af06c3 P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:10.336139 11033 tablet_copy_client.cc:670] T 6f5e370b77cc42fdb0b886ea19af06c3 P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:10.336330 8060 consensus_queue.cc:237] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:10.338541 9842 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:10.339103 10874 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:10.341651 11033 tablet_copy_client.cc:538] T 6f5e370b77cc42fdb0b886ea19af06c3 P dede81a8c0164427a61532d01b6fc157: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:10.352807 11033 tablet_bootstrap.cc:492] T 6f5e370b77cc42fdb0b886ea19af06c3 P dede81a8c0164427a61532d01b6fc157: Bootstrap starting.
I20250627 13:54:10.356606 10104 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:10.356905 7932 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:10.357592 11193 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:10.358340 10874 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:10.360718 11193 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:10.363883 10104 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:10.367524 11024 ts_tablet_manager.cc:927] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b: Initiating tablet copy from peer a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:54:10.368108 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet d2eb537d49804e1b8ec5a217b1564988 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:10.368530 9841 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:10.373193 11194 ts_tablet_manager.cc:1905] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:10.372992 7653 catalog_manager.cc:5582] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 reported cstate change: config changed from index 7 to 8, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:10.377504 7782 tablet_service.cc:1515] Processing DeleteTablet for tablet d2eb537d49804e1b8ec5a217b1564988 with delete_type TABLET_DATA_TOMBSTONED (TS 83574c1bf8ea45adaff939158f93c096 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:45826
I20250627 13:54:10.379408 10933 raft_consensus.cc:1062] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER dede81a8c0164427a61532d01b6fc157 to VOTER
I20250627 13:54:10.380568 10933 consensus_queue.cc:237] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } }
I20250627 13:54:10.383406 7932 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:10.398286 11024 tablet_copy_client.cc:323] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:43675
I20250627 13:54:10.402175 8192 consensus_queue.cc:237] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
W20250627 13:54:10.405125 8126 consensus_peers.cc:487] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14dba72d45694defb83f41c1572722ca. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:10.405411 10104 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.405987 10583 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:10.410094 7932 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.415452 7800 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.417086 10583 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:10.419515 11034 raft_consensus.cc:1025] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: attempt to promote peer dede81a8c0164427a61532d01b6fc157: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250627 13:54:10.419975 11051 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:10.421784 11034 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:10.423182 7929 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:10.426527 7801 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:10.432421 7654 catalog_manager.cc:5582] T 14dba72d45694defb83f41c1572722ca P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 4 to 5, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:10.432639 10104 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:10.436173 8060 consensus_queue.cc:237] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:10.438344 10102 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:10.439033 10713 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:10.443775 9842 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:10.443862 11189 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:10.444381 10713 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:10.445948 9842 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:10.448254 10102 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:10.456681 11034 raft_consensus.cc:1062] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e: attempting to promote NON_VOTER dede81a8c0164427a61532d01b6fc157 to VOTER
I20250627 13:54:10.457592 11034 consensus_queue.cc:237] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
W20250627 13:54:10.461238 8126 consensus_peers.cc:487] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14dba72d45694defb83f41c1572722ca. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:10.462085 7801 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:10.462740 11048 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:10.467008 8060 consensus_queue.cc:237] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:10.467769 10103 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:10.470065 10583 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:10.472321 11051 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:10.473037 10102 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:10.481390 7658 catalog_manager.cc:5582] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b reported cstate change: config changed from index 8 to 9, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:10.485698 11098 tablet_bootstrap.cc:492] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:10.486196 11098 tablet_bootstrap.cc:492] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe: Bootstrap complete.
I20250627 13:54:10.486493 11098 ts_tablet_manager.cc:1397] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe: Time spent bootstrapping tablet: real 0.181s user 0.051s sys 0.006s
I20250627 13:54:10.487514 11098 raft_consensus.cc:357] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:10.487708 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet d2eb537d49804e1b8ec5a217b1564988 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:10.487802 11098 raft_consensus.cc:738] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6686c6c365bf4f4582058890161676fe, State: Initialized, Role: LEARNER
I20250627 13:54:10.488005 11098 consensus_queue.cc:260] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:10.488595 11098 ts_tablet_manager.cc:1428] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250627 13:54:10.491212 7801 raft_consensus.cc:1273] T 91664403eed74badbeba83d69f597e1f P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.491386 7929 raft_consensus.cc:1273] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.491791 10713 consensus_queue.cc:1035] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:10.493506 11188 consensus_queue.cc:1035] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:10.494089 10102 raft_consensus.cc:1273] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.495674 11188 consensus_queue.cc:1035] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:10.496717 7911 tablet_service.cc:1515] Processing DeleteTablet for tablet d2eb537d49804e1b8ec5a217b1564988 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 8 (current committed config index is 9)) from {username='slave'} at 127.0.0.1:59302
I20250627 13:54:10.507612 7652 catalog_manager.cc:5582] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 reported cstate change: config changed from index 5 to 6, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:10.520452 11188 raft_consensus.cc:2953] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:10.524995 7802 raft_consensus.cc:1273] T 5184c3f8e465472e9a98c14f8e703415 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:10.525131 8192 raft_consensus.cc:1273] T 5184c3f8e465472e9a98c14f8e703415 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:10.525978 10933 consensus_queue.cc:1035] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:10.526507 10713 consensus_queue.cc:1035] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:10.528687 10104 raft_consensus.cc:2953] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:10.531632 7953 tablet_copy_service.cc:140] P a41a89ae8ed743edb2b5cf4490d8c854: Received BeginTabletCopySession request for tablet 03640a5124d2405faf4a26d0e22e1ffc from peer 956ff058e6d64579a1963aabdb6d466b ({username='slave'} at 127.0.103.70:60171)
I20250627 13:54:10.531812 7953 tablet_copy_service.cc:161] P a41a89ae8ed743edb2b5cf4490d8c854: Beginning new tablet copy session on tablet 03640a5124d2405faf4a26d0e22e1ffc from peer 956ff058e6d64579a1963aabdb6d466b at {username='slave'} at 127.0.103.70:60171: session id = 956ff058e6d64579a1963aabdb6d466b-03640a5124d2405faf4a26d0e22e1ffc
I20250627 13:54:10.531998 7800 raft_consensus.cc:2953] T 91664403eed74badbeba83d69f597e1f P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:10.534569 7953 tablet_copy_source_session.cc:215] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:10.534965 11189 raft_consensus.cc:2953] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:10.535662 11110 ts_tablet_manager.cc:927] T 03640a5124d2405faf4a26d0e22e1ffc P 2a1282ebe92a4c9a92eed3b13f69012a: Initiating tablet copy from peer a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:54:10.535600 7802 raft_consensus.cc:2953] T 5184c3f8e465472e9a98c14f8e703415 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:10.537029 11024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03640a5124d2405faf4a26d0e22e1ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:10.537297 7659 catalog_manager.cc:5582] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 4 to 5, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:10.542264 7932 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:10.542850 10583 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:10.544611 7932 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:10.545217 11033 tablet_bootstrap.cc:492] T 6f5e370b77cc42fdb0b886ea19af06c3 P dede81a8c0164427a61532d01b6fc157: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:10.545682 11033 tablet_bootstrap.cc:492] T 6f5e370b77cc42fdb0b886ea19af06c3 P dede81a8c0164427a61532d01b6fc157: Bootstrap complete.
I20250627 13:54:10.546025 11033 ts_tablet_manager.cc:1397] T 6f5e370b77cc42fdb0b886ea19af06c3 P dede81a8c0164427a61532d01b6fc157: Time spent bootstrapping tablet: real 0.193s user 0.047s sys 0.004s
I20250627 13:54:10.546859 11033 raft_consensus.cc:357] T 6f5e370b77cc42fdb0b886ea19af06c3 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:10.547062 11033 raft_consensus.cc:738] T 6f5e370b77cc42fdb0b886ea19af06c3 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dede81a8c0164427a61532d01b6fc157, State: Initialized, Role: LEARNER
I20250627 13:54:10.547264 11033 consensus_queue.cc:260] T 6f5e370b77cc42fdb0b886ea19af06c3 P dede81a8c0164427a61532d01b6fc157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:10.547775 11033 ts_tablet_manager.cc:1428] T 6f5e370b77cc42fdb0b886ea19af06c3 P dede81a8c0164427a61532d01b6fc157: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250627 13:54:10.548974 7802 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:10.549554 8213 tablet_copy_service.cc:342] P e4c831df4da841929d13d6c783099b0e: Request end of tablet copy session dede81a8c0164427a61532d01b6fc157-6f5e370b77cc42fdb0b886ea19af06c3 received from {username='slave'} at 127.0.103.72:33995
I20250627 13:54:10.549722 8213 tablet_copy_service.cc:434] P e4c831df4da841929d13d6c783099b0e: ending tablet copy session dede81a8c0164427a61532d01b6fc157-6f5e370b77cc42fdb0b886ea19af06c3 on tablet 6f5e370b77cc42fdb0b886ea19af06c3 with peer dede81a8c0164427a61532d01b6fc157
I20250627 13:54:10.551578 7929 raft_consensus.cc:2953] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:10.552585 11024 tablet_copy_client.cc:806] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 0 data blocks...
W20250627 13:54:10.554208 7999 consensus_peers.cc:487] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 6686c6c365bf4f4582058890161676fe (127.0.103.71:37859): Couldn't send request to peer 6686c6c365bf4f4582058890161676fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5184c3f8e465472e9a98c14f8e703415. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:10.554561 11024 tablet_copy_client.cc:670] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:10.555688 7659 catalog_manager.cc:5582] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index -1 to 4, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:10.556671 11024 tablet_copy_client.cc:538] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:10.558900 7822 tablet_copy_service.cc:342] P 83574c1bf8ea45adaff939158f93c096: Request end of tablet copy session 6686c6c365bf4f4582058890161676fe-40431e3bd7244904a4e65f59477bd21d received from {username='slave'} at 127.0.103.71:53529
I20250627 13:54:10.559044 7822 tablet_copy_service.cc:434] P 83574c1bf8ea45adaff939158f93c096: ending tablet copy session 6686c6c365bf4f4582058890161676fe-40431e3bd7244904a4e65f59477bd21d on tablet 40431e3bd7244904a4e65f59477bd21d with peer 6686c6c365bf4f4582058890161676fe
I20250627 13:54:10.558780 11224 tablet_replica.cc:331] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854: stopping tablet replica
I20250627 13:54:10.561367 11123 raft_consensus.cc:1062] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096: attempting to promote NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a to VOTER
I20250627 13:54:10.562417 11123 consensus_queue.cc:237] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:10.566579 11024 tablet_bootstrap.cc:492] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b: Bootstrap starting.
I20250627 13:54:10.568516 11224 raft_consensus.cc:2241] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:10.568985 11224 raft_consensus.cc:2270] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:10.577306 8191 raft_consensus.cc:2953] T 5184c3f8e465472e9a98c14f8e703415 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:10.578130 11224 ts_tablet_manager.cc:1905] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:10.582187 8061 consensus_queue.cc:237] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:10.598951 11194 ts_tablet_manager.cc:1918] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250627 13:54:10.601370 9711 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.601460 10961 tablet_bootstrap.cc:492] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:10.603015 11110 tablet_copy_client.cc:323] T 03640a5124d2405faf4a26d0e22e1ffc P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:43675
I20250627 13:54:10.602784 10926 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:10.604771 10988 tablet_bootstrap.cc:492] T 96fe0309adc54fb6a426bae02f26f898 P dede81a8c0164427a61532d01b6fc157: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250627 13:54:10.605219 7953 tablet_copy_service.cc:140] P a41a89ae8ed743edb2b5cf4490d8c854: Received BeginTabletCopySession request for tablet 03640a5124d2405faf4a26d0e22e1ffc from peer 2a1282ebe92a4c9a92eed3b13f69012a ({username='slave'} at 127.0.103.69:49083)
I20250627 13:54:10.605302 10988 tablet_bootstrap.cc:492] T 96fe0309adc54fb6a426bae02f26f898 P dede81a8c0164427a61532d01b6fc157: Bootstrap complete.
I20250627 13:54:10.605374 7953 tablet_copy_service.cc:161] P a41a89ae8ed743edb2b5cf4490d8c854: Beginning new tablet copy session on tablet 03640a5124d2405faf4a26d0e22e1ffc from peer 2a1282ebe92a4c9a92eed3b13f69012a at {username='slave'} at 127.0.103.69:49083: session id = 2a1282ebe92a4c9a92eed3b13f69012a-03640a5124d2405faf4a26d0e22e1ffc
I20250627 13:54:10.605567 10988 ts_tablet_manager.cc:1397] T 96fe0309adc54fb6a426bae02f26f898 P dede81a8c0164427a61532d01b6fc157: Time spent bootstrapping tablet: real 0.406s user 0.043s sys 0.005s
I20250627 13:54:10.607374 8060 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.608414 7953 tablet_copy_source_session.cc:215] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:10.608713 10930 raft_consensus.cc:1062] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096: attempting to promote NON_VOTER 6686c6c365bf4f4582058890161676fe to VOTER
I20250627 13:54:10.609560 10930 consensus_queue.cc:237] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:10.610248 10961 tablet_bootstrap.cc:492] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b: Bootstrap complete.
I20250627 13:54:10.610529 10961 ts_tablet_manager.cc:1397] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b: Time spent bootstrapping tablet: real 0.728s user 0.045s sys 0.011s
I20250627 13:54:10.611402 10961 raft_consensus.cc:357] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:10.612112 10928 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:10.624251 10961 raft_consensus.cc:738] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 956ff058e6d64579a1963aabdb6d466b, State: Initialized, Role: LEARNER
I20250627 13:54:10.626174 10961 consensus_queue.cc:260] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:10.627646 10988 raft_consensus.cc:357] T 96fe0309adc54fb6a426bae02f26f898 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:10.631415 10961 ts_tablet_manager.cc:1428] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b: Time spent starting tablet: real 0.021s user 0.001s sys 0.002s
I20250627 13:54:10.634817 10928 raft_consensus.cc:1062] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096: attempting to promote NON_VOTER 6686c6c365bf4f4582058890161676fe to VOTER
I20250627 13:54:10.635618 10928 consensus_queue.cc:237] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:10.638108 8213 tablet_copy_service.cc:342] P e4c831df4da841929d13d6c783099b0e: Request end of tablet copy session 956ff058e6d64579a1963aabdb6d466b-da90822ff15e4e5da05114ef38d3970a received from {username='slave'} at 127.0.103.70:33381
I20250627 13:54:10.638281 8213 tablet_copy_service.cc:434] P e4c831df4da841929d13d6c783099b0e: ending tablet copy session 956ff058e6d64579a1963aabdb6d466b-da90822ff15e4e5da05114ef38d3970a on tablet da90822ff15e4e5da05114ef38d3970a with peer 956ff058e6d64579a1963aabdb6d466b
I20250627 13:54:10.641520 10988 raft_consensus.cc:738] T 96fe0309adc54fb6a426bae02f26f898 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dede81a8c0164427a61532d01b6fc157, State: Initialized, Role: LEARNER
I20250627 13:54:10.644186 10988 consensus_queue.cc:260] T 96fe0309adc54fb6a426bae02f26f898 P dede81a8c0164427a61532d01b6fc157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:10.646342 10926 raft_consensus.cc:1062] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096: attempting to promote NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a to VOTER
I20250627 13:54:10.647064 10926 consensus_queue.cc:237] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:10.654816 10988 ts_tablet_manager.cc:1428] T 96fe0309adc54fb6a426bae02f26f898 P dede81a8c0164427a61532d01b6fc157: Time spent starting tablet: real 0.049s user 0.004s sys 0.000s
I20250627 13:54:10.655462 8083 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session dede81a8c0164427a61532d01b6fc157-96fe0309adc54fb6a426bae02f26f898 received from {username='slave'} at 127.0.103.72:50607
I20250627 13:54:10.655611 8083 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session dede81a8c0164427a61532d01b6fc157-96fe0309adc54fb6a426bae02f26f898 on tablet 96fe0309adc54fb6a426bae02f26f898 with peer dede81a8c0164427a61532d01b6fc157
W20250627 13:54:10.663117 7996 consensus_peers.cc:487] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e2bd060a4a9e4765b45b0242ae56e489. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:10.663337 7800 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:10.663899 11189 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:10.664516 8192 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:10.665232 11189 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:10.673681 11098 ts_tablet_manager.cc:927] T 96fe0309adc54fb6a426bae02f26f898 P 6686c6c365bf4f4582058890161676fe: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:10.676782 7802 consensus_queue.cc:237] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:10.676977 11188 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:10.678428 7800 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:10.687438 11056 tablet_bootstrap.cc:492] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:10.688874 7658 catalog_manager.cc:5582] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:10.697290 7929 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.702723 9842 raft_consensus.cc:1215] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250627 13:54:10.707784 11056 tablet_bootstrap.cc:492] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap complete.
I20250627 13:54:10.708134 11056 ts_tablet_manager.cc:1397] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a: Time spent bootstrapping tablet: real 0.383s user 0.046s sys 0.009s
I20250627 13:54:10.708973 11056 raft_consensus.cc:357] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:10.709185 11056 raft_consensus.cc:738] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a1282ebe92a4c9a92eed3b13f69012a, State: Initialized, Role: LEARNER
I20250627 13:54:10.712004 11110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03640a5124d2405faf4a26d0e22e1ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:10.712415 11056 consensus_queue.cc:260] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:10.715103 11056 ts_tablet_manager.cc:1428] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20250627 13:54:10.715857 8083 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session 2a1282ebe92a4c9a92eed3b13f69012a-eed3236c4fe8455cb9527b9f4b754e81 received from {username='slave'} at 127.0.103.69:59515
I20250627 13:54:10.716006 8083 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session 2a1282ebe92a4c9a92eed3b13f69012a-eed3236c4fe8455cb9527b9f4b754e81 on tablet eed3236c4fe8455cb9527b9f4b754e81 with peer 2a1282ebe92a4c9a92eed3b13f69012a
I20250627 13:54:10.720266 8192 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:10.724406 11258 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:10.729910 9974 raft_consensus.cc:1215] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250627 13:54:10.739301 10933 raft_consensus.cc:1062] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER dede81a8c0164427a61532d01b6fc157 to VOTER
I20250627 13:54:10.740427 10933 consensus_queue.cc:237] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } }
I20250627 13:54:10.756354 8061 raft_consensus.cc:1273] T c322cfebf12c4fef8d88fbd68e7e8b6f P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.756558 7932 raft_consensus.cc:1273] T c322cfebf12c4fef8d88fbd68e7e8b6f P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.756798 7931 raft_consensus.cc:1273] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.756934 11123 consensus_queue.cc:1035] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:10.757690 7931 raft_consensus.cc:1273] T 720cd645f0b7414f819fcbff3fd262c3 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.758287 11194 log.cc:1199] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/wals/e3053b529a264e8697a1df9e17eeea81
I20250627 13:54:10.758745 10104 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.759325 11189 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:10.759775 11194 tablet_replica.cc:331] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: stopping tablet replica
I20250627 13:54:10.760077 11194 raft_consensus.cc:2241] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:10.761848 8192 raft_consensus.cc:1273] T ff83fdbab2b9417eac50495c54f0da54 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.766733 7645 catalog_manager.cc:4928] TS 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467): tablet e3053b529a264e8697a1df9e17eeea81 (table rebalance_test_table_0 [id=d8bfc9268a4a4c2c92bc20d56ac73086]) successfully deleted
I20250627 13:54:10.767693 8192 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:10.768930 10926 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:10.768599 11272 consensus_queue.cc:1035] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20250627 13:54:10.768795 11273 consensus_queue.cc:1035] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20250627 13:54:10.771802 11258 consensus_queue.cc:1035] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:10.774226 9974 raft_consensus.cc:1273] T 720cd645f0b7414f819fcbff3fd262c3 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.774456 9973 raft_consensus.cc:1273] T ff83fdbab2b9417eac50495c54f0da54 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.776722 8191 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.777426 11189 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:10.788781 7929 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.789314 10933 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:10.795853 10632 raft_consensus.cc:1062] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854: attempting to promote NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a to VOTER
I20250627 13:54:10.797122 10632 consensus_queue.cc:237] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:10.797902 11273 consensus_queue.cc:1035] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:10.804121 11098 tablet_copy_client.cc:323] T 96fe0309adc54fb6a426bae02f26f898 P 6686c6c365bf4f4582058890161676fe: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:10.820729 9711 raft_consensus.cc:1273] T c322cfebf12c4fef8d88fbd68e7e8b6f P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.820552 11123 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:10.821538 11189 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:10.826857 8193 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:10.829845 11194 pending_rounds.cc:70] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: Trying to abort 1 pending ops.
I20250627 13:54:10.830078 11194 pending_rounds.cc:77] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: Aborting op as it isn't in flight: id { term: 1 index: 7 } timestamp: 7172228916105973760 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "d2eb537d49804e1b8ec5a217b1564988" old_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } } new_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } } }
I20250627 13:54:10.830430 11194 raft_consensus.cc:2915] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Aborting config change with OpId 1.7: Aborted: Op aborted
I20250627 13:54:10.830557 11194 raft_consensus.cc:2887] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250627 13:54:10.830287 7658 catalog_manager.cc:5582] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 4 to 5, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:10.830682 11194 raft_consensus.cc:2270] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:10.830044 7929 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:10.831753 11194 ts_tablet_manager.cc:1905] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:10.833220 8061 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:10.836521 11274 consensus_queue.cc:1035] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.076s
I20250627 13:54:10.838088 11269 consensus_queue.cc:1035] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:10.842845 8060 consensus_queue.cc:237] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } }
I20250627 13:54:10.844311 7658 catalog_manager.cc:5582] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:10.845975 10104 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:10.850258 7931 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:10.857911 11272 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:10.869174 8190 raft_consensus.cc:1273] T 720cd645f0b7414f819fcbff3fd262c3 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.869808 11257 consensus_queue.cc:1035] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:10.870474 7802 consensus_queue.cc:237] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:10.871039 9711 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:10.873624 8191 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:10.879386 10933 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:10.881748 11189 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:10.887493 7802 raft_consensus.cc:1273] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.887698 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 644985e3086d4614b6598a727cb60c1d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:10.887822 11273 consensus_queue.cc:1035] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:10.888587 11273 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:10.888715 11258 raft_consensus.cc:2953] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:10.889139 9710 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:10.890275 10928 raft_consensus.cc:2953] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:10.892153 7658 catalog_manager.cc:5582] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 5 to 6, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:10.893246 10103 raft_consensus.cc:1215] T 6f5e370b77cc42fdb0b886ea19af06c3 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250627 13:54:10.896641 10102 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:10.898399 8060 raft_consensus.cc:2953] T c322cfebf12c4fef8d88fbd68e7e8b6f P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:10.898636 7659 catalog_manager.cc:5582] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:10.902211 11188 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:10.904188 10102 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:10.905937 8190 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:10.913661 10930 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:10.915045 8083 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet 96fe0309adc54fb6a426bae02f26f898 from peer 6686c6c365bf4f4582058890161676fe ({username='slave'} at 127.0.103.71:59981)
I20250627 13:54:10.915228 8083 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet 96fe0309adc54fb6a426bae02f26f898 from peer 6686c6c365bf4f4582058890161676fe at {username='slave'} at 127.0.103.71:59981: session id = 6686c6c365bf4f4582058890161676fe-96fe0309adc54fb6a426bae02f26f898
I20250627 13:54:10.919319 9710 raft_consensus.cc:2953] T c322cfebf12c4fef8d88fbd68e7e8b6f P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:10.920054 8083 tablet_copy_source_session.cc:215] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:10.923858 7659 catalog_manager.cc:5582] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:10.925706 9974 raft_consensus.cc:2953] T ff83fdbab2b9417eac50495c54f0da54 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:10.934818 10874 raft_consensus.cc:1062] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER 956ff058e6d64579a1963aabdb6d466b to VOTER
I20250627 13:54:10.935505 11098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96fe0309adc54fb6a426bae02f26f898. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:10.935555 10874 consensus_queue.cc:237] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:10.938825 8190 raft_consensus.cc:1273] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.940598 9842 raft_consensus.cc:1273] T 570b134177314b31851951c411a9f4bf P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.940104 8190 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:10.941586 11188 consensus_queue.cc:1035] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:10.941874 11274 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:10.942339 11188 consensus_queue.cc:1035] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:10.944566 7802 consensus_queue.cc:237] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:10.944372 8188 raft_consensus.cc:2953] T ff83fdbab2b9417eac50495c54f0da54 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:10.951378 9711 raft_consensus.cc:1273] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:10.956054 9711 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:10.957553 7652 catalog_manager.cc:5582] T ff83fdbab2b9417eac50495c54f0da54 P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:10.962417 9710 raft_consensus.cc:1273] T c322cfebf12c4fef8d88fbd68e7e8b6f P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:10.968976 10928 consensus_queue.cc:1035] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:10.977496 11189 raft_consensus.cc:2953] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:10.979970 11321 ts_tablet_manager.cc:927] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:10.982404 9842 raft_consensus.cc:2953] T 570b134177314b31851951c411a9f4bf P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:10.983970 7658 catalog_manager.cc:5582] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: config changed from index 5 to 6, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:10.984167 7659 catalog_manager.cc:5582] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 4 to 5, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:10.986212 8189 raft_consensus.cc:2953] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:10.990183 8041 tablet_service.cc:1515] Processing DeleteTablet for tablet 87726824b05a4b968386f57d3a7a4aad with delete_type TABLET_DATA_TOMBSTONED (TS 9cf479bdcd0242738d2a77db4ad9f660 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59760
W20250627 13:54:10.996640 7737 consensus_peers.cc:487] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 -> Peer dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213): Couldn't send request to peer dede81a8c0164427a61532d01b6fc157. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 897206e017ff4305a8cc9092e30f46b6. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:10.997499 10961 ts_tablet_manager.cc:927] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: Initiating tablet copy from peer e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:54:11.003516 8060 consensus_queue.cc:237] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:11.005661 9842 raft_consensus.cc:1273] T 570b134177314b31851951c411a9f4bf P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:11.006312 11189 consensus_queue.cc:1035] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20250627 13:54:11.007642 9907 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletCopyService.BeginTabletCopySession -> {remote=127.0.103.67:38933, user_credentials={real_user=slave}} blocked reactor thread for 72578.2us
I20250627 13:54:11.009277 7911 tablet_service.cc:1515] Processing DeleteTablet for tablet 644985e3086d4614b6598a727cb60c1d with delete_type TABLET_DATA_TOMBSTONED (TS a41a89ae8ed743edb2b5cf4490d8c854 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59302
I20250627 13:54:11.009128 11189 raft_consensus.cc:2953] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:11.009899 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:11.010229 7929 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:11.015522 7656 catalog_manager.cc:5582] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 5 to 6, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:11.015723 7932 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:11.020200 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 570b134177314b31851951c411a9f4bf with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:11.022243 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:11.027499 7930 raft_consensus.cc:1273] T 570b134177314b31851951c411a9f4bf P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:11.027846 11021 consensus_queue.cc:1035] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20250627 13:54:11.030159 9841 raft_consensus.cc:2953] T 570b134177314b31851951c411a9f4bf P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:11.029127 7927 raft_consensus.cc:2953] T c322cfebf12c4fef8d88fbd68e7e8b6f P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:11.033110 7802 consensus_queue.cc:237] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:11.034265 11266 raft_consensus.cc:2953] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:11.031181 7928 raft_consensus.cc:2953] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:11.053326 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 87726824b05a4b968386f57d3a7a4aad with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:11.058585 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:11.066771 10102 raft_consensus.cc:1215] T 96fe0309adc54fb6a426bae02f26f898 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20250627 13:54:11.067065 10102 raft_consensus.cc:2953] T 96fe0309adc54fb6a426bae02f26f898 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:11.069434 11110 tablet_copy_client.cc:806] T 03640a5124d2405faf4a26d0e22e1ffc P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:11.071796 7933 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:11.078154 11110 tablet_copy_client.cc:670] T 03640a5124d2405faf4a26d0e22e1ffc P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:11.082741 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:11.084864 7928 raft_consensus.cc:2953] T 720cd645f0b7414f819fcbff3fd262c3 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:11.090214 11194 ts_tablet_manager.cc:1918] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20250627 13:54:11.090467 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:11.091418 8193 raft_consensus.cc:1273] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:11.092247 11189 consensus_queue.cc:1035] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:11.094099 8193 raft_consensus.cc:2953] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:11.094666 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:11.102486 7658 catalog_manager.cc:5582] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:11.103331 7926 raft_consensus.cc:1273] T c322cfebf12c4fef8d88fbd68e7e8b6f P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:11.108285 8193 raft_consensus.cc:2953] T 720cd645f0b7414f819fcbff3fd262c3 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:11.109445 11098 tablet_copy_client.cc:806] T 96fe0309adc54fb6a426bae02f26f898 P 6686c6c365bf4f4582058890161676fe: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:11.109750 11098 tablet_copy_client.cc:670] T 96fe0309adc54fb6a426bae02f26f898 P 6686c6c365bf4f4582058890161676fe: tablet copy: Starting download of 1 WAL segments...
W20250627 13:54:11.114774 7866 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.70:40647, user_credentials={real_user=slave}} blocked reactor thread for 85759.6us
I20250627 13:54:11.116910 11321 tablet_copy_client.cc:323] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:11.119499 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:11.109113 7800 consensus_queue.cc:237] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:11.125501 11194 log.cc:1199] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/wals/d2eb537d49804e1b8ec5a217b1564988
I20250627 13:54:11.126816 7932 raft_consensus.cc:1273] T 720cd645f0b7414f819fcbff3fd262c3 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:11.127283 11265 consensus_queue.cc:1035] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:11.110100 11269 raft_consensus.cc:2953] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:11.116393 10928 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:11.129106 8189 consensus_queue.cc:237] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:11.121183 11263 consensus_queue.cc:1035] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:11.136132 10961 tablet_copy_client.cc:323] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.68:46139
I20250627 13:54:11.136740 8213 tablet_copy_service.cc:140] P e4c831df4da841929d13d6c783099b0e: Received BeginTabletCopySession request for tablet 14dba72d45694defb83f41c1572722ca from peer 956ff058e6d64579a1963aabdb6d466b ({username='slave'} at 127.0.103.70:33381)
I20250627 13:54:11.136916 8213 tablet_copy_service.cc:161] P e4c831df4da841929d13d6c783099b0e: Beginning new tablet copy session on tablet 14dba72d45694defb83f41c1572722ca from peer 956ff058e6d64579a1963aabdb6d466b at {username='slave'} at 127.0.103.70:33381: session id = 956ff058e6d64579a1963aabdb6d466b-14dba72d45694defb83f41c1572722ca
I20250627 13:54:11.137871 11263 raft_consensus.cc:2953] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:11.140331 8213 tablet_copy_source_session.cc:215] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:11.143267 7932 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:11.144357 8043 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:11.148828 7926 raft_consensus.cc:2953] T 720cd645f0b7414f819fcbff3fd262c3 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:11.153214 7927 raft_consensus.cc:2953] T c322cfebf12c4fef8d88fbd68e7e8b6f P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:11.157375 7658 catalog_manager.cc:5582] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 5 to 6, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:11.160283 10988 ts_tablet_manager.cc:927] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157: Initiating tablet copy from peer 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:54:11.163136 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:11.164064 10961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14dba72d45694defb83f41c1572722ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:11.169014 8043 tablet_service.cc:1515] Processing DeleteTablet for tablet c322cfebf12c4fef8d88fbd68e7e8b6f with delete_type TABLET_DATA_TOMBSTONED (TS 9cf479bdcd0242738d2a77db4ad9f660 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59760
I20250627 13:54:11.170632 11339 ts_tablet_manager.cc:927] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:11.174019 11339 tablet_copy_client.cc:323] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:11.175125 11331 tablet_replica.cc:331] T 87726824b05a4b968386f57d3a7a4aad P 9cf479bdcd0242738d2a77db4ad9f660: stopping tablet replica
I20250627 13:54:11.175256 7658 catalog_manager.cc:5582] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 5 to 6, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:11.175743 11331 raft_consensus.cc:2241] T 87726824b05a4b968386f57d3a7a4aad P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:11.186435 8171 tablet_service.cc:1515] Processing DeleteTablet for tablet 720cd645f0b7414f819fcbff3fd262c3 with delete_type TABLET_DATA_TOMBSTONED (TS e4c831df4da841929d13d6c783099b0e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48616
W20250627 13:54:11.186734 7738 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.69:33101, user_credentials={real_user=slave}} blocked reactor thread for 77308.4us
I20250627 13:54:11.187234 7645 catalog_manager.cc:4928] TS 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467): tablet d2eb537d49804e1b8ec5a217b1564988 (table rebalance_test_table_0 [id=d8bfc9268a4a4c2c92bc20d56ac73086]) successfully deleted
I20250627 13:54:11.189626 10988 tablet_copy_client.cc:323] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:37467
I20250627 13:54:11.193485 7931 raft_consensus.cc:1273] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:11.194200 11272 consensus_queue.cc:1035] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20250627 13:54:11.176153 7866 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.67:38933, user_credentials={real_user=slave}} blocked reactor thread for 61141.9us
I20250627 13:54:11.197957 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet c322cfebf12c4fef8d88fbd68e7e8b6f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:11.198328 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 720cd645f0b7414f819fcbff3fd262c3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:11.199878 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:11.200083 9711 raft_consensus.cc:1215] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250627 13:54:11.206573 11224 ts_tablet_manager.cc:1918] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20250627 13:54:11.206818 11224 log.cc:1199] T d2eb537d49804e1b8ec5a217b1564988 P a41a89ae8ed743edb2b5cf4490d8c854: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/wals/d2eb537d49804e1b8ec5a217b1564988
I20250627 13:54:11.207798 11224 tablet_replica.cc:331] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854: stopping tablet replica
I20250627 13:54:11.211421 11269 raft_consensus.cc:2953] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:11.213487 11110 tablet_copy_client.cc:538] T 03640a5124d2405faf4a26d0e22e1ffc P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:11.216382 8193 raft_consensus.cc:1273] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:11.217231 11110 tablet_bootstrap.cc:492] T 03640a5124d2405faf4a26d0e22e1ffc P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap starting.
I20250627 13:54:11.218523 7911 tablet_service.cc:1515] Processing DeleteTablet for tablet 570b134177314b31851951c411a9f4bf with delete_type TABLET_DATA_TOMBSTONED (TS a41a89ae8ed743edb2b5cf4490d8c854 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59302
I20250627 13:54:11.222149 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet ff83fdbab2b9417eac50495c54f0da54 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:11.222903 11343 tablet_replica.cc:331] T 720cd645f0b7414f819fcbff3fd262c3 P e4c831df4da841929d13d6c783099b0e: stopping tablet replica
I20250627 13:54:11.223515 7659 catalog_manager.cc:5582] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 5 to 6, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:11.224537 11331 raft_consensus.cc:2270] T 87726824b05a4b968386f57d3a7a4aad P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:11.225086 7931 raft_consensus.cc:2953] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:11.176559 8083 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet e2bd060a4a9e4765b45b0242ae56e489 from peer 956ff058e6d64579a1963aabdb6d466b ({username='slave'} at 127.0.103.70:59981)
I20250627 13:54:11.225855 8083 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet e2bd060a4a9e4765b45b0242ae56e489 from peer 956ff058e6d64579a1963aabdb6d466b at {username='slave'} at 127.0.103.70:59981: session id = 956ff058e6d64579a1963aabdb6d466b-e2bd060a4a9e4765b45b0242ae56e489
I20250627 13:54:11.228700 8083 tablet_copy_source_session.cc:215] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:11.228899 11021 consensus_queue.cc:1035] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:11.230093 11021 consensus_queue.cc:1035] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:11.231613 7647 catalog_manager.cc:4928] TS a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675): tablet d2eb537d49804e1b8ec5a217b1564988 (table rebalance_test_table_0 [id=d8bfc9268a4a4c2c92bc20d56ac73086]) successfully deleted
I20250627 13:54:11.238517 11339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2bd060a4a9e4765b45b0242ae56e489. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:11.238759 11331 ts_tablet_manager.cc:1905] T 87726824b05a4b968386f57d3a7a4aad P 9cf479bdcd0242738d2a77db4ad9f660: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:11.242424 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet 5184c3f8e465472e9a98c14f8e703415 from peer 6686c6c365bf4f4582058890161676fe ({username='slave'} at 127.0.103.71:59981)
I20250627 13:54:11.242578 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet 5184c3f8e465472e9a98c14f8e703415 from peer 6686c6c365bf4f4582058890161676fe at {username='slave'} at 127.0.103.71:59981: session id = 6686c6c365bf4f4582058890161676fe-5184c3f8e465472e9a98c14f8e703415
I20250627 13:54:11.243144 11224 raft_consensus.cc:2241] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:11.245270 8082 tablet_copy_source_session.cc:215] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:11.246464 9974 raft_consensus.cc:1273] T ff83fdbab2b9417eac50495c54f0da54 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:11.247205 11098 tablet_copy_client.cc:538] T 96fe0309adc54fb6a426bae02f26f898 P 6686c6c365bf4f4582058890161676fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:11.246889 9973 raft_consensus.cc:2953] T 720cd645f0b7414f819fcbff3fd262c3 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:11.247639 11274 consensus_queue.cc:1035] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:11.251225 9972 raft_consensus.cc:1273] T 720cd645f0b7414f819fcbff3fd262c3 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:11.237790 8171 tablet_service.cc:1515] Processing DeleteTablet for tablet ff83fdbab2b9417eac50495c54f0da54 with delete_type TABLET_DATA_TOMBSTONED (TS e4c831df4da841929d13d6c783099b0e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48616
I20250627 13:54:11.251976 11098 tablet_bootstrap.cc:492] T 96fe0309adc54fb6a426bae02f26f898 P 6686c6c365bf4f4582058890161676fe: Bootstrap starting.
I20250627 13:54:11.251952 11343 raft_consensus.cc:2241] T 720cd645f0b7414f819fcbff3fd262c3 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:11.252250 11343 raft_consensus.cc:2270] T 720cd645f0b7414f819fcbff3fd262c3 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:11.252465 11321 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5184c3f8e465472e9a98c14f8e703415. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:11.263453 11224 raft_consensus.cc:2270] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:11.266139 11274 consensus_queue.cc:1035] T 720cd645f0b7414f819fcbff3fd262c3 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20250627 13:54:11.267081 8128 consensus_peers.cc:487] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e -> Peer dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213): Couldn't send request to peer dede81a8c0164427a61532d01b6fc157. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da90822ff15e4e5da05114ef38d3970a. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:11.267526 9841 raft_consensus.cc:1273] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:11.268151 11034 consensus_queue.cc:1035] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:11.268272 11224 ts_tablet_manager.cc:1905] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:11.271196 7932 raft_consensus.cc:1273] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:11.274232 11034 consensus_queue.cc:1035] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:11.274999 10084 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:11.277706 11343 ts_tablet_manager.cc:1905] T 720cd645f0b7414f819fcbff3fd262c3 P e4c831df4da841929d13d6c783099b0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:11.281724 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:11.297453 8189 consensus_queue.cc:237] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } }
I20250627 13:54:11.298349 7801 raft_consensus.cc:1273] T da90822ff15e4e5da05114ef38d3970a P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:11.299113 11034 consensus_queue.cc:1035] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:11.301306 9972 raft_consensus.cc:2953] T 720cd645f0b7414f819fcbff3fd262c3 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:11.302435 9973 raft_consensus.cc:2953] T ff83fdbab2b9417eac50495c54f0da54 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:11.310286 11021 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:11.312088 7802 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:11.312696 11034 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:11.318720 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:11.318905 8189 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:11.338851 9711 raft_consensus.cc:2953] T c322cfebf12c4fef8d88fbd68e7e8b6f P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:11.361827 7658 catalog_manager.cc:5582] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:11.364135 11051 raft_consensus.cc:1025] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: attempt to promote peer 956ff058e6d64579a1963aabdb6d466b: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250627 13:54:11.369115 11024 tablet_bootstrap.cc:492] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:11.374492 7822 tablet_copy_service.cc:140] P 83574c1bf8ea45adaff939158f93c096: Received BeginTabletCopySession request for tablet 897206e017ff4305a8cc9092e30f46b6 from peer dede81a8c0164427a61532d01b6fc157 ({username='slave'} at 127.0.103.72:38213)
I20250627 13:54:11.374691 7822 tablet_copy_service.cc:161] P 83574c1bf8ea45adaff939158f93c096: Beginning new tablet copy session on tablet 897206e017ff4305a8cc9092e30f46b6 from peer dede81a8c0164427a61532d01b6fc157 at {username='slave'} at 127.0.103.72:38213: session id = dede81a8c0164427a61532d01b6fc157-897206e017ff4305a8cc9092e30f46b6
I20250627 13:54:11.378521 7822 tablet_copy_source_session.cc:215] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:11.380283 11024 tablet_bootstrap.cc:492] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b: Bootstrap complete.
I20250627 13:54:11.380546 11024 ts_tablet_manager.cc:1397] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b: Time spent bootstrapping tablet: real 0.814s user 0.045s sys 0.013s
I20250627 13:54:11.381328 11024 raft_consensus.cc:357] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:11.381533 11024 raft_consensus.cc:738] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 956ff058e6d64579a1963aabdb6d466b, State: Initialized, Role: LEARNER
I20250627 13:54:11.381780 11024 consensus_queue.cc:260] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:11.382663 11024 ts_tablet_manager.cc:1428] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:54:11.383354 7953 tablet_copy_service.cc:342] P a41a89ae8ed743edb2b5cf4490d8c854: Request end of tablet copy session 956ff058e6d64579a1963aabdb6d466b-03640a5124d2405faf4a26d0e22e1ffc received from {username='slave'} at 127.0.103.70:60171
I20250627 13:54:11.383513 7953 tablet_copy_service.cc:434] P a41a89ae8ed743edb2b5cf4490d8c854: ending tablet copy session 956ff058e6d64579a1963aabdb6d466b-03640a5124d2405faf4a26d0e22e1ffc on tablet 03640a5124d2405faf4a26d0e22e1ffc with peer 956ff058e6d64579a1963aabdb6d466b
I20250627 13:54:11.387709 9709 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:11.394497 10988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 897206e017ff4305a8cc9092e30f46b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:11.394825 11339 tablet_copy_client.cc:806] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:11.395043 11339 tablet_copy_client.cc:670] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:11.399845 11339 tablet_copy_client.cc:538] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:11.401170 10988 tablet_copy_client.cc:806] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:11.401486 10988 tablet_copy_client.cc:670] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:11.403517 11339 tablet_bootstrap.cc:492] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap starting.
I20250627 13:54:11.404815 10084 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:11.408067 10988 tablet_copy_client.cc:538] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:11.411827 10988 tablet_bootstrap.cc:492] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157: Bootstrap starting.
I20250627 13:54:11.413452 10102 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:11.414064 11034 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:11.416918 11034 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:11.418445 10104 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:11.425516 7648 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 14dba72d45694defb83f41c1572722ca with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:11.428565 7652 catalog_manager.cc:5582] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 6 to 7, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:11.428751 7802 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:11.462595 11034 raft_consensus.cc:1062] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e: attempting to promote NON_VOTER dede81a8c0164427a61532d01b6fc157 to VOTER
I20250627 13:54:11.463727 11034 consensus_queue.cc:237] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } }
I20250627 13:54:11.470811 9822 tablet_service.cc:1515] Processing DeleteTablet for tablet 14dba72d45694defb83f41c1572722ca with delete_type TABLET_DATA_TOMBSTONED (TS 956ff058e6d64579a1963aabdb6d466b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55228
I20250627 13:54:11.481868 10104 raft_consensus.cc:1273] T 6f5e370b77cc42fdb0b886ea19af06c3 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:11.482720 11218 consensus_queue.cc:1035] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:11.494741 8060 raft_consensus.cc:1273] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:11.495520 11218 consensus_queue.cc:1035] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:11.500660 11098 tablet_bootstrap.cc:492] T 96fe0309adc54fb6a426bae02f26f898 P 6686c6c365bf4f4582058890161676fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:11.507189 11098 tablet_bootstrap.cc:492] T 96fe0309adc54fb6a426bae02f26f898 P 6686c6c365bf4f4582058890161676fe: Bootstrap complete.
I20250627 13:54:11.507498 11098 ts_tablet_manager.cc:1397] T 96fe0309adc54fb6a426bae02f26f898 P 6686c6c365bf4f4582058890161676fe: Time spent bootstrapping tablet: real 0.256s user 0.047s sys 0.002s
I20250627 13:54:11.509658 11098 raft_consensus.cc:357] T 96fe0309adc54fb6a426bae02f26f898 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:11.509975 11098 raft_consensus.cc:738] T 96fe0309adc54fb6a426bae02f26f898 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6686c6c365bf4f4582058890161676fe, State: Initialized, Role: LEARNER
I20250627 13:54:11.510236 11098 consensus_queue.cc:260] T 96fe0309adc54fb6a426bae02f26f898 P 6686c6c365bf4f4582058890161676fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:11.510880 11274 raft_consensus.cc:1062] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096: attempting to promote NON_VOTER 6686c6c365bf4f4582058890161676fe to VOTER
I20250627 13:54:11.510990 11098 ts_tablet_manager.cc:1428] T 96fe0309adc54fb6a426bae02f26f898 P 6686c6c365bf4f4582058890161676fe: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250627 13:54:11.511705 11274 consensus_queue.cc:237] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
W20250627 13:54:11.513991 8126 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250627 13:54:11.524597 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session 6686c6c365bf4f4582058890161676fe-96fe0309adc54fb6a426bae02f26f898 received from {username='slave'} at 127.0.103.71:59981
I20250627 13:54:11.524763 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session 6686c6c365bf4f4582058890161676fe-96fe0309adc54fb6a426bae02f26f898 on tablet 96fe0309adc54fb6a426bae02f26f898 with peer 6686c6c365bf4f4582058890161676fe
I20250627 13:54:11.527128 7802 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:11.533838 11339 tablet_bootstrap.cc:492] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:11.534463 11339 tablet_bootstrap.cc:492] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap complete.
I20250627 13:54:11.534956 11339 ts_tablet_manager.cc:1397] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b: Time spent bootstrapping tablet: real 0.132s user 0.055s sys 0.003s
I20250627 13:54:11.536063 11339 raft_consensus.cc:357] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:11.536460 11339 raft_consensus.cc:738] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 956ff058e6d64579a1963aabdb6d466b, State: Initialized, Role: LEARNER
I20250627 13:54:11.536778 11339 consensus_queue.cc:260] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:11.537583 11339 ts_tablet_manager.cc:1428] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:54:11.539417 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session 956ff058e6d64579a1963aabdb6d466b-e2bd060a4a9e4765b45b0242ae56e489 received from {username='slave'} at 127.0.103.70:59981
I20250627 13:54:11.539597 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session 956ff058e6d64579a1963aabdb6d466b-e2bd060a4a9e4765b45b0242ae56e489 on tablet e2bd060a4a9e4765b45b0242ae56e489 with peer 956ff058e6d64579a1963aabdb6d466b
I20250627 13:54:11.541703 7926 raft_consensus.cc:1273] T 6f5e370b77cc42fdb0b886ea19af06c3 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:11.542232 7933 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:11.547430 11218 raft_consensus.cc:2953] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:11.554816 11188 raft_consensus.cc:1062] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER dede81a8c0164427a61532d01b6fc157 to VOTER
I20250627 13:54:11.556016 11188 consensus_queue.cc:237] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:11.565623 9841 raft_consensus.cc:2953] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:11.567873 7658 catalog_manager.cc:5582] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 4 to 5, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:11.575999 10583 raft_consensus.cc:1062] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e: attempting to promote NON_VOTER 956ff058e6d64579a1963aabdb6d466b to VOTER
I20250627 13:54:11.577093 10583 consensus_queue.cc:237] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:11.578923 9973 raft_consensus.cc:1273] T 96fe0309adc54fb6a426bae02f26f898 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:11.579599 10874 consensus_queue.cc:1035] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:11.580721 9842 raft_consensus.cc:1273] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:11.581638 11051 consensus_queue.cc:1035] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:11.583280 7802 raft_consensus.cc:1273] T da90822ff15e4e5da05114ef38d3970a P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:11.584077 11051 consensus_queue.cc:1035] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:11.585733 10104 raft_consensus.cc:1273] T 96fe0309adc54fb6a426bae02f26f898 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:11.585443 7800 raft_consensus.cc:2953] T da90822ff15e4e5da05114ef38d3970a P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:11.586283 10874 consensus_queue.cc:1035] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:11.598902 11034 raft_consensus.cc:2953] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
W20250627 13:54:11.605547 8128 consensus_peers.cc:487] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e -> Peer dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213): Couldn't send request to peer dede81a8c0164427a61532d01b6fc157. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da90822ff15e4e5da05114ef38d3970a. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:11.612229 11051 consensus_queue.cc:1035] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:11.612826 11051 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:11.612902 10961 tablet_copy_client.cc:806] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:11.613206 10961 tablet_copy_client.cc:670] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 1 WAL segments...
W20250627 13:54:11.628015 7645 catalog_manager.cc:4908] TS 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): delete failed for tablet 14dba72d45694defb83f41c1572722ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14dba72d45694defb83f41c1572722ca already in progress: copying tablet
I20250627 13:54:11.642830 11321 tablet_copy_client.cc:806] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:11.643088 11321 tablet_copy_client.cc:670] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:11.640467 7931 consensus_queue.cc:237] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:11.652246 9709 raft_consensus.cc:1273] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:11.652693 11321 tablet_copy_client.cc:538] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:11.652884 8189 raft_consensus.cc:1273] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:11.653752 11335 consensus_queue.cc:1035] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:11.654072 11336 consensus_queue.cc:1035] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:11.656351 8189 raft_consensus.cc:1273] T 40431e3bd7244904a4e65f59477bd21d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:11.657933 11335 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:11.658396 8193 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:11.658461 9709 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:11.662304 7647 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet b265cfda4bfe48908659208ec8eeab2d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:11.664350 9973 raft_consensus.cc:1273] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:11.664592 11274 consensus_queue.cc:1035] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:11.668394 11331 ts_tablet_manager.cc:1918] T 87726824b05a4b968386f57d3a7a4aad P 9cf479bdcd0242738d2a77db4ad9f660: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250627 13:54:11.672874 11273 consensus_queue.cc:1035] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:11.614917 9842 raft_consensus.cc:1215] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20250627 13:54:11.698307 11273 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:11.702992 8193 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:11.703245 9973 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:11.711220 11110 tablet_bootstrap.cc:492] T 03640a5124d2405faf4a26d0e22e1ffc P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:11.711737 11110 tablet_bootstrap.cc:492] T 03640a5124d2405faf4a26d0e22e1ffc P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap complete.
I20250627 13:54:11.712054 11110 ts_tablet_manager.cc:1397] T 03640a5124d2405faf4a26d0e22e1ffc P 2a1282ebe92a4c9a92eed3b13f69012a: Time spent bootstrapping tablet: real 0.495s user 0.101s sys 0.126s
I20250627 13:54:11.713093 11110 raft_consensus.cc:357] T 03640a5124d2405faf4a26d0e22e1ffc P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:11.713354 11110 raft_consensus.cc:738] T 03640a5124d2405faf4a26d0e22e1ffc P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a1282ebe92a4c9a92eed3b13f69012a, State: Initialized, Role: LEARNER
I20250627 13:54:11.713666 11110 consensus_queue.cc:260] T 03640a5124d2405faf4a26d0e22e1ffc P 2a1282ebe92a4c9a92eed3b13f69012a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:11.715773 11343 ts_tablet_manager.cc:1918] T 720cd645f0b7414f819fcbff3fd262c3 P e4c831df4da841929d13d6c783099b0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250627 13:54:11.715977 11343 log.cc:1199] T 720cd645f0b7414f819fcbff3fd262c3 P e4c831df4da841929d13d6c783099b0e: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal/wals/720cd645f0b7414f819fcbff3fd262c3
I20250627 13:54:11.716168 7932 raft_consensus.cc:1273] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:11.716966 11343 tablet_replica.cc:331] T ff83fdbab2b9417eac50495c54f0da54 P e4c831df4da841929d13d6c783099b0e: stopping tablet replica
W20250627 13:54:11.717015 8127 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250627 13:54:11.717027 7648 catalog_manager.cc:4928] TS e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): tablet 720cd645f0b7414f819fcbff3fd262c3 (table rebalance_test_table_6 [id=f83cab74056a4e7681a701169d1c1716]) successfully deleted
I20250627 13:54:11.717278 11343 raft_consensus.cc:2241] T ff83fdbab2b9417eac50495c54f0da54 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:11.717542 11343 raft_consensus.cc:2270] T ff83fdbab2b9417eac50495c54f0da54 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:11.718194 11321 tablet_bootstrap.cc:492] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe: Bootstrap starting.
I20250627 13:54:11.721366 9842 raft_consensus.cc:2953] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:11.721982 11110 ts_tablet_manager.cc:1428] T 03640a5124d2405faf4a26d0e22e1ffc P 2a1282ebe92a4c9a92eed3b13f69012a: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20250627 13:54:11.722064 7801 raft_consensus.cc:2953] T da90822ff15e4e5da05114ef38d3970a P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:11.726831 7927 raft_consensus.cc:1273] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:11.728389 11218 consensus_queue.cc:1035] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:11.729493 7953 tablet_copy_service.cc:342] P a41a89ae8ed743edb2b5cf4490d8c854: Request end of tablet copy session 2a1282ebe92a4c9a92eed3b13f69012a-03640a5124d2405faf4a26d0e22e1ffc received from {username='slave'} at 127.0.103.69:49083
I20250627 13:54:11.729702 7953 tablet_copy_service.cc:434] P a41a89ae8ed743edb2b5cf4490d8c854: ending tablet copy session 2a1282ebe92a4c9a92eed3b13f69012a-03640a5124d2405faf4a26d0e22e1ffc on tablet 03640a5124d2405faf4a26d0e22e1ffc with peer 2a1282ebe92a4c9a92eed3b13f69012a
W20250627 13:54:11.718062 7998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.72:35213, user_credentials={real_user=slave}} blocked reactor thread for 129234us
I20250627 13:54:11.721745 11188 raft_consensus.cc:1025] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: attempt to promote peer 6686c6c365bf4f4582058890161676fe: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250627 13:54:11.733723 11331 log.cc:1199] T 87726824b05a4b968386f57d3a7a4aad P 9cf479bdcd0242738d2a77db4ad9f660: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal/wals/87726824b05a4b968386f57d3a7a4aad
I20250627 13:54:11.733412 7927 raft_consensus.cc:2953] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:11.734084 11051 raft_consensus.cc:2953] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:11.734894 11331 tablet_replica.cc:331] T c322cfebf12c4fef8d88fbd68e7e8b6f P 9cf479bdcd0242738d2a77db4ad9f660: stopping tablet replica
I20250627 13:54:11.734988 7645 catalog_manager.cc:4928] TS 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933): tablet 87726824b05a4b968386f57d3a7a4aad (table rebalance_test_table_1 [id=8961aa0df2f1488f82cd55cfa22ad214]) successfully deleted
I20250627 13:54:11.735172 11331 raft_consensus.cc:2241] T c322cfebf12c4fef8d88fbd68e7e8b6f P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:11.737421 9842 raft_consensus.cc:1215] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250627 13:54:11.739702 8061 raft_consensus.cc:2953] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:11.741452 11331 raft_consensus.cc:2270] T c322cfebf12c4fef8d88fbd68e7e8b6f P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:11.741091 10103 raft_consensus.cc:2953] T 6f5e370b77cc42fdb0b886ea19af06c3 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:11.742430 11331 ts_tablet_manager.cc:1905] T c322cfebf12c4fef8d88fbd68e7e8b6f P 9cf479bdcd0242738d2a77db4ad9f660: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:11.745675 7655 catalog_manager.cc:5582] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 4 to 5, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:11.754825 11343 ts_tablet_manager.cc:1905] T ff83fdbab2b9417eac50495c54f0da54 P e4c831df4da841929d13d6c783099b0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:11.759022 7656 catalog_manager.cc:5582] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe reported cstate change: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:11.763444 7658 catalog_manager.cc:5582] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: config changed from index 5 to 6, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:11.767678 11343 ts_tablet_manager.cc:1918] T ff83fdbab2b9417eac50495c54f0da54 P e4c831df4da841929d13d6c783099b0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250627 13:54:11.768980 7782 tablet_service.cc:1515] Processing DeleteTablet for tablet b265cfda4bfe48908659208ec8eeab2d with delete_type TABLET_DATA_TOMBSTONED (TS 83574c1bf8ea45adaff939158f93c096 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45826
I20250627 13:54:11.772810 11343 log.cc:1199] T ff83fdbab2b9417eac50495c54f0da54 P e4c831df4da841929d13d6c783099b0e: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal/wals/ff83fdbab2b9417eac50495c54f0da54
I20250627 13:54:11.773940 7653 catalog_manager.cc:5582] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b reported cstate change: config changed from index 5 to 6, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:11.775476 7648 catalog_manager.cc:4928] TS e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): tablet ff83fdbab2b9417eac50495c54f0da54 (table rebalance_test_table_8 [id=b84f3fe3fd7c4f01902547d2db982f90]) successfully deleted
I20250627 13:54:11.782650 11389 tablet_replica.cc:331] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096: stopping tablet replica
I20250627 13:54:11.782866 11389 raft_consensus.cc:2241] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:11.783097 11389 raft_consensus.cc:2270] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:11.783533 8193 consensus_queue.cc:237] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } }
I20250627 13:54:11.785866 10961 tablet_copy_client.cc:538] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:11.786031 8061 raft_consensus.cc:1273] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:11.787436 10103 raft_consensus.cc:1273] T 6f5e370b77cc42fdb0b886ea19af06c3 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:11.787941 10583 consensus_queue.cc:1035] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:11.788357 10583 consensus_queue.cc:1035] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:11.789096 11331 ts_tablet_manager.cc:1918] T c322cfebf12c4fef8d88fbd68e7e8b6f P 9cf479bdcd0242738d2a77db4ad9f660: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250627 13:54:11.789229 11331 log.cc:1199] T c322cfebf12c4fef8d88fbd68e7e8b6f P 9cf479bdcd0242738d2a77db4ad9f660: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal/wals/c322cfebf12c4fef8d88fbd68e7e8b6f
I20250627 13:54:11.791126 10583 raft_consensus.cc:2953] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:11.791870 8061 raft_consensus.cc:2953] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:11.796077 7648 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 6f5e370b77cc42fdb0b886ea19af06c3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:11.797523 10102 raft_consensus.cc:2953] T 6f5e370b77cc42fdb0b886ea19af06c3 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:11.799999 7654 catalog_manager.cc:5582] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 5 to 6, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:11.801286 11389 ts_tablet_manager.cc:1905] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:11.809372 7645 catalog_manager.cc:4928] TS 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933): tablet c322cfebf12c4fef8d88fbd68e7e8b6f (table rebalance_test_table_5 [id=dd79cdfc35164ae6a649d3a77ee397c2]) successfully deleted
I20250627 13:54:11.818935 7911 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f5e370b77cc42fdb0b886ea19af06c3 with delete_type TABLET_DATA_TOMBSTONED (TS a41a89ae8ed743edb2b5cf4490d8c854 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59302
I20250627 13:54:11.824775 11389 ts_tablet_manager.cc:1918] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250627 13:54:11.825237 11389 log.cc:1199] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/wals/b265cfda4bfe48908659208ec8eeab2d
I20250627 13:54:11.826472 7645 catalog_manager.cc:4928] TS 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467): tablet b265cfda4bfe48908659208ec8eeab2d (table rebalance_test_table_0 [id=d8bfc9268a4a4c2c92bc20d56ac73086]) successfully deleted
I20250627 13:54:11.834537 7927 raft_consensus.cc:2953] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:11.842007 7933 raft_consensus.cc:1273] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:11.843081 11375 consensus_queue.cc:1035] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:11.852262 11321 tablet_bootstrap.cc:492] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:11.852701 11321 tablet_bootstrap.cc:492] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe: Bootstrap complete.
I20250627 13:54:11.857949 11321 ts_tablet_manager.cc:1397] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe: Time spent bootstrapping tablet: real 0.200s user 0.048s sys 0.000s
I20250627 13:54:11.859012 11321 raft_consensus.cc:357] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:11.859344 11321 raft_consensus.cc:738] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6686c6c365bf4f4582058890161676fe, State: Initialized, Role: LEARNER
I20250627 13:54:11.859571 11321 consensus_queue.cc:260] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:11.860842 11321 ts_tablet_manager.cc:1428] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250627 13:54:11.871256 11224 ts_tablet_manager.cc:1918] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250627 13:54:11.888036 11272 consensus_queue.cc:1035] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:11.895774 10988 tablet_bootstrap.cc:492] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:11.896168 10988 tablet_bootstrap.cc:492] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157: Bootstrap complete.
I20250627 13:54:11.896422 10988 ts_tablet_manager.cc:1397] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157: Time spent bootstrapping tablet: real 0.485s user 0.046s sys 0.000s
I20250627 13:54:11.897025 10988 raft_consensus.cc:357] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:11.897203 10988 raft_consensus.cc:738] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dede81a8c0164427a61532d01b6fc157, State: Initialized, Role: LEARNER
I20250627 13:54:11.897408 10988 consensus_queue.cc:260] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:11.898017 8193 raft_consensus.cc:1273] T 96fe0309adc54fb6a426bae02f26f898 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:11.904970 10988 ts_tablet_manager.cc:1428] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20250627 13:54:11.908412 7822 tablet_copy_service.cc:342] P 83574c1bf8ea45adaff939158f93c096: Request end of tablet copy session dede81a8c0164427a61532d01b6fc157-897206e017ff4305a8cc9092e30f46b6 received from {username='slave'} at 127.0.103.72:38213
I20250627 13:54:11.908555 7822 tablet_copy_service.cc:434] P 83574c1bf8ea45adaff939158f93c096: ending tablet copy session dede81a8c0164427a61532d01b6fc157-897206e017ff4305a8cc9092e30f46b6 on tablet 897206e017ff4305a8cc9092e30f46b6 with peer dede81a8c0164427a61532d01b6fc157
I20250627 13:54:11.912146 11375 raft_consensus.cc:2953] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:11.913714 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session 6686c6c365bf4f4582058890161676fe-5184c3f8e465472e9a98c14f8e703415 received from {username='slave'} at 127.0.103.71:59981
I20250627 13:54:11.913856 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session 6686c6c365bf4f4582058890161676fe-5184c3f8e465472e9a98c14f8e703415 on tablet 5184c3f8e465472e9a98c14f8e703415 with peer 6686c6c365bf4f4582058890161676fe
I20250627 13:54:11.915123 10933 consensus_queue.cc:1035] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:11.915486 11189 raft_consensus.cc:1062] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a to VOTER
I20250627 13:54:11.916334 11189 consensus_queue.cc:237] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:11.920827 10103 raft_consensus.cc:2953] T 96fe0309adc54fb6a426bae02f26f898 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:11.929819 7654 catalog_manager.cc:5582] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 5 to 6, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:11.932032 10961 tablet_bootstrap.cc:492] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: Bootstrap starting.
I20250627 13:54:11.934003 8193 raft_consensus.cc:2953] T 96fe0309adc54fb6a426bae02f26f898 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:11.938838 9973 raft_consensus.cc:2953] T 96fe0309adc54fb6a426bae02f26f898 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:11.941918 9709 raft_consensus.cc:1273] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:11.942009 7801 raft_consensus.cc:1273] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:11.942633 11378 consensus_queue.cc:1035] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:11.943042 11382 consensus_queue.cc:1035] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:11.943554 10933 raft_consensus.cc:1062] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER 6686c6c365bf4f4582058890161676fe to VOTER
I20250627 13:54:11.944373 10933 consensus_queue.cc:237] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:11.947539 9973 raft_consensus.cc:1273] T 96fe0309adc54fb6a426bae02f26f898 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:11.947626 8193 raft_consensus.cc:1273] T 96fe0309adc54fb6a426bae02f26f898 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:11.948422 10102 raft_consensus.cc:1273] T 96fe0309adc54fb6a426bae02f26f898 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:11.889618 11021 raft_consensus.cc:491] T 570b134177314b31851951c411a9f4bf P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9cf479bdcd0242738d2a77db4ad9f660)
I20250627 13:54:11.949335 11021 raft_consensus.cc:513] T 570b134177314b31851951c411a9f4bf P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:11.949422 11382 consensus_queue.cc:1035] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:11.950294 11021 leader_election.cc:290] T 570b134177314b31851951c411a9f4bf P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:11.950690 11224 log.cc:1199] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/wals/644985e3086d4614b6598a727cb60c1d
I20250627 13:54:11.950738 8061 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "570b134177314b31851951c411a9f4bf" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660" is_pre_election: true
I20250627 13:54:11.955889 11224 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59302 (request call id 24) took 1043 ms. Trace:
I20250627 13:54:11.956048 11382 consensus_queue.cc:1035] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:11.956386 8193 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "570b134177314b31851951c411a9f4bf" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e4c831df4da841929d13d6c783099b0e" is_pre_election: true
I20250627 13:54:11.956846 11382 consensus_queue.cc:1035] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:11.925859 7927 raft_consensus.cc:2953] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:11.939414 7931 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:11.940559 7932 raft_consensus.cc:1273] T eed3236c4fe8455cb9527b9f4b754e81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:11.944051 7926 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:11.956959 7869 leader_election.cc:304] T 570b134177314b31851951c411a9f4bf P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a41a89ae8ed743edb2b5cf4490d8c854; no voters: 9cf479bdcd0242738d2a77db4ad9f660, e4c831df4da841929d13d6c783099b0e
I20250627 13:54:11.914861 7933 consensus_queue.cc:237] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:11.959786 11021 raft_consensus.cc:2747] T 570b134177314b31851951c411a9f4bf P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:54:11.963840 11378 raft_consensus.cc:2953] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:11.964646 9973 raft_consensus.cc:2953] T 96fe0309adc54fb6a426bae02f26f898 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:11.971252 8193 raft_consensus.cc:2953] T 96fe0309adc54fb6a426bae02f26f898 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:11.972828 10102 raft_consensus.cc:2953] T 96fe0309adc54fb6a426bae02f26f898 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:11.975170 7652 catalog_manager.cc:5582] T 96fe0309adc54fb6a426bae02f26f898 P 6686c6c365bf4f4582058890161676fe reported cstate change: config changed from index 6 to 7, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:11.977391 11382 consensus_queue.cc:1035] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:11.981570 11335 raft_consensus.cc:1062] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854: attempting to promote NON_VOTER 956ff058e6d64579a1963aabdb6d466b to VOTER
I20250627 13:54:11.982504 11335 consensus_queue.cc:237] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:11.983085 8061 consensus_queue.cc:237] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:11.987829 9973 raft_consensus.cc:1273] T 96fe0309adc54fb6a426bae02f26f898 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:11.988004 8193 raft_consensus.cc:1273] T 96fe0309adc54fb6a426bae02f26f898 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:11.957839 11224 rpcz_store.cc:276] 0627 13:54:10.912515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0627 13:54:10.912577 (+ 62us) service_pool.cc:225] Handling call
0627 13:54:11.207666 (+295089us) ts_tablet_manager.cc:1145] Deleting tablet 644985e3086d4614b6598a727cb60c1d
0627 13:54:11.207676 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0627 13:54:11.617905 (+410229us) tablet_metadata.cc:641] Metadata flushed
0627 13:54:11.871061 (+253156us) tablet_metadata.cc:641] Metadata flushed
0627 13:54:11.955811 (+ 84750us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17657,"tablet-delete.queue_time_us":198192}
I20250627 13:54:11.988417 11188 consensus_queue.cc:1035] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:11.988549 7647 catalog_manager.cc:4928] TS a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675): tablet 644985e3086d4614b6598a727cb60c1d (table rebalance_test_table_1 [id=8961aa0df2f1488f82cd55cfa22ad214]) successfully deleted
I20250627 13:54:11.990069 11188 consensus_queue.cc:1035] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:11.990914 11224 tablet_replica.cc:331] T 570b134177314b31851951c411a9f4bf P a41a89ae8ed743edb2b5cf4490d8c854: stopping tablet replica
I20250627 13:54:11.991123 11224 raft_consensus.cc:2241] T 570b134177314b31851951c411a9f4bf P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:11.993584 10874 raft_consensus.cc:2953] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:11.998020 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 96fe0309adc54fb6a426bae02f26f898 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:12.000258 9973 raft_consensus.cc:2953] T 96fe0309adc54fb6a426bae02f26f898 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:12.000380 7653 catalog_manager.cc:5582] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 7 to 8, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:12.003794 11224 raft_consensus.cc:2270] T 570b134177314b31851951c411a9f4bf P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:12.004544 11224 ts_tablet_manager.cc:1905] T 570b134177314b31851951c411a9f4bf P a41a89ae8ed743edb2b5cf4490d8c854: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:12.004913 7911 tablet_service.cc:1515] Processing DeleteTablet for tablet 96fe0309adc54fb6a426bae02f26f898 with delete_type TABLET_DATA_TOMBSTONED (TS a41a89ae8ed743edb2b5cf4490d8c854 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:59302
I20250627 13:54:12.009287 8061 consensus_queue.cc:237] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:12.010434 7930 raft_consensus.cc:1273] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:12.012492 8193 raft_consensus.cc:2953] T 96fe0309adc54fb6a426bae02f26f898 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:12.014317 9973 raft_consensus.cc:1273] T 96fe0309adc54fb6a426bae02f26f898 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:12.015244 10874 consensus_queue.cc:1035] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:12.018306 8189 raft_consensus.cc:1273] T 96fe0309adc54fb6a426bae02f26f898 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:12.019451 11378 consensus_queue.cc:1035] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:12.019726 11188 raft_consensus.cc:2953] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:12.022334 10988 ts_tablet_manager.cc:927] T da90822ff15e4e5da05114ef38d3970a P dede81a8c0164427a61532d01b6fc157: Initiating tablet copy from peer e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:54:12.022994 10988 tablet_copy_client.cc:323] T da90822ff15e4e5da05114ef38d3970a P dede81a8c0164427a61532d01b6fc157: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.68:46139
I20250627 13:54:12.024230 8213 tablet_copy_service.cc:140] P e4c831df4da841929d13d6c783099b0e: Received BeginTabletCopySession request for tablet da90822ff15e4e5da05114ef38d3970a from peer dede81a8c0164427a61532d01b6fc157 ({username='slave'} at 127.0.103.72:33995)
I20250627 13:54:12.024399 8213 tablet_copy_service.cc:161] P e4c831df4da841929d13d6c783099b0e: Beginning new tablet copy session on tablet da90822ff15e4e5da05114ef38d3970a from peer dede81a8c0164427a61532d01b6fc157 at {username='slave'} at 127.0.103.72:33995: session id = dede81a8c0164427a61532d01b6fc157-da90822ff15e4e5da05114ef38d3970a
I20250627 13:54:12.027565 8213 tablet_copy_source_session.cc:215] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:12.028990 10988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da90822ff15e4e5da05114ef38d3970a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:12.030208 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 96fe0309adc54fb6a426bae02f26f898 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:12.031489 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:12.032939 7659 catalog_manager.cc:5582] T 96fe0309adc54fb6a426bae02f26f898 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 8 to 9, VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:12.033543 9973 raft_consensus.cc:2953] T 96fe0309adc54fb6a426bae02f26f898 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:12.033990 8189 raft_consensus.cc:2953] T 96fe0309adc54fb6a426bae02f26f898 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:12.038524 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:12.039105 10988 tablet_copy_client.cc:806] T da90822ff15e4e5da05114ef38d3970a P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:12.039386 10988 tablet_copy_client.cc:670] T da90822ff15e4e5da05114ef38d3970a P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:12.040632 10103 raft_consensus.cc:1273] T 96fe0309adc54fb6a426bae02f26f898 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:12.052208 10988 tablet_copy_client.cc:538] T da90822ff15e4e5da05114ef38d3970a P dede81a8c0164427a61532d01b6fc157: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:12.055882 11224 ts_tablet_manager.cc:1918] T 570b134177314b31851951c411a9f4bf P a41a89ae8ed743edb2b5cf4490d8c854: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20250627 13:54:12.056061 11224 log.cc:1199] T 570b134177314b31851951c411a9f4bf P a41a89ae8ed743edb2b5cf4490d8c854: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/wals/570b134177314b31851951c411a9f4bf
I20250627 13:54:12.056201 10988 tablet_bootstrap.cc:492] T da90822ff15e4e5da05114ef38d3970a P dede81a8c0164427a61532d01b6fc157: Bootstrap starting.
I20250627 13:54:12.056919 7647 catalog_manager.cc:4928] TS a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675): tablet 570b134177314b31851951c411a9f4bf (table rebalance_test_table_7 [id=46b91111142d4e388d01cd06d3d41940]) successfully deleted
I20250627 13:54:12.057330 11224 tablet_replica.cc:331] T 6f5e370b77cc42fdb0b886ea19af06c3 P a41a89ae8ed743edb2b5cf4490d8c854: stopping tablet replica
I20250627 13:54:12.057528 11224 raft_consensus.cc:2241] T 6f5e370b77cc42fdb0b886ea19af06c3 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:12.057749 11224 pending_rounds.cc:70] T 6f5e370b77cc42fdb0b886ea19af06c3 P a41a89ae8ed743edb2b5cf4490d8c854: Trying to abort 1 pending ops.
I20250627 13:54:12.057958 11224 pending_rounds.cc:77] T 6f5e370b77cc42fdb0b886ea19af06c3 P a41a89ae8ed743edb2b5cf4490d8c854: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7172228921191329792 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "6f5e370b77cc42fdb0b886ea19af06c3" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } } }
I20250627 13:54:12.058276 11224 raft_consensus.cc:2915] T 6f5e370b77cc42fdb0b886ea19af06c3 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20250627 13:54:12.058458 11224 raft_consensus.cc:2887] T 6f5e370b77cc42fdb0b886ea19af06c3 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250627 13:54:12.058604 11224 raft_consensus.cc:2270] T 6f5e370b77cc42fdb0b886ea19af06c3 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:12.059839 11224 ts_tablet_manager.cc:1905] T 6f5e370b77cc42fdb0b886ea19af06c3 P a41a89ae8ed743edb2b5cf4490d8c854: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:12.062678 9842 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:12.062738 8061 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:12.063180 9709 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:12.063580 11029 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:12.063884 11335 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:12.064062 11029 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:12.064275 8189 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:12.066555 11188 raft_consensus.cc:2953] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:12.071926 7800 raft_consensus.cc:2953] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:12.074951 11335 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:12.075013 7927 raft_consensus.cc:2953] T eed3236c4fe8455cb9527b9f4b754e81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:12.079263 9711 raft_consensus.cc:2953] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:12.083402 7653 catalog_manager.cc:5582] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:12.099013 11335 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:12.105291 9709 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:12.108206 11029 raft_consensus.cc:1062] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854: attempting to promote NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a to VOTER
I20250627 13:54:12.109874 11029 consensus_queue.cc:237] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:12.114500 11224 ts_tablet_manager.cc:1918] T 6f5e370b77cc42fdb0b886ea19af06c3 P a41a89ae8ed743edb2b5cf4490d8c854: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250627 13:54:12.114615 11224 log.cc:1199] T 6f5e370b77cc42fdb0b886ea19af06c3 P a41a89ae8ed743edb2b5cf4490d8c854: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/wals/6f5e370b77cc42fdb0b886ea19af06c3
I20250627 13:54:12.115310 11224 tablet_replica.cc:331] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854: stopping tablet replica
I20250627 13:54:12.115794 8193 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20250627 13:54:12.116614 7869 consensus_peers.cc:487] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 -> Peer 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101): Couldn't send request to peer 2a1282ebe92a4c9a92eed3b13f69012a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72db4bf98ab04b7189640f658d8fdcd7. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:12.116948 11021 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:12.116839 7653 catalog_manager.cc:5582] T 03640a5124d2405faf4a26d0e22e1ffc P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: config changed from index 5 to 6, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:12.118148 8193 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:12.119367 9711 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:12.120890 11021 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:12.124387 11224 raft_consensus.cc:2241] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:12.124727 11224 raft_consensus.cc:2270] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:12.128788 11224 ts_tablet_manager.cc:1905] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:12.136900 8190 consensus_queue.cc:237] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:12.138731 8060 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:12.141902 9842 raft_consensus.cc:1273] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:12.142868 7800 raft_consensus.cc:1273] T da90822ff15e4e5da05114ef38d3970a P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:12.143937 10583 consensus_queue.cc:1035] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:12.144402 10583 consensus_queue.cc:1035] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:12.147204 9841 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:12.149334 7647 catalog_manager.cc:4928] TS a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675): tablet 6f5e370b77cc42fdb0b886ea19af06c3 (table rebalance_test_table_3 [id=ec6a9ed6ab4d416988b2abe2f8d29c55]) successfully deleted
I20250627 13:54:12.150027 8062 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:12.150785 8063 raft_consensus.cc:1273] T 72db4bf98ab04b7189640f658d8fdcd7 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:12.153124 7801 raft_consensus.cc:1273] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:12.156021 7933 raft_consensus.cc:1273] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:12.156419 9973 raft_consensus.cc:1215] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250627 13:54:12.158670 11034 consensus_queue.cc:1035] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:12.159629 11335 consensus_queue.cc:1035] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:12.161391 11224 ts_tablet_manager.cc:1918] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20250627 13:54:12.162264 7800 raft_consensus.cc:2953] T da90822ff15e4e5da05114ef38d3970a P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:12.161585 11335 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:12.161942 11029 consensus_queue.cc:1035] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:12.164832 11224 log.cc:1199] T 96fe0309adc54fb6a426bae02f26f898 P a41a89ae8ed743edb2b5cf4490d8c854: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/wals/96fe0309adc54fb6a426bae02f26f898
I20250627 13:54:12.161314 11034 raft_consensus.cc:2953] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:12.177886 11029 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:12.180545 8061 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:12.184702 9840 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:12.184990 8189 consensus_queue.cc:237] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } }
I20250627 13:54:12.186940 8193 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:12.189453 7933 raft_consensus.cc:2953] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:12.189723 11335 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:12.191972 7652 catalog_manager.cc:5582] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index 6 to 7, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:12.192374 9841 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:12.198256 10103 raft_consensus.cc:1215] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250627 13:54:12.199782 10103 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:12.200354 10583 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:12.202690 10583 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:12.205862 7648 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet da90822ff15e4e5da05114ef38d3970a with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:12.206637 9840 raft_consensus.cc:2953] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:12.207495 7647 catalog_manager.cc:4928] TS a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675): tablet 96fe0309adc54fb6a426bae02f26f898 (table rebalance_test_table_9 [id=19e5686b50fe4187a733aa6413ef2ce0]) successfully deleted
I20250627 13:54:12.208439 7648 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 14dba72d45694defb83f41c1572722ca with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:12.209192 10103 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:12.210278 7652 catalog_manager.cc:5582] T da90822ff15e4e5da05114ef38d3970a P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 6 to 7, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) evicted. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:12.214802 10084 tablet_service.cc:1515] Processing DeleteTablet for tablet 96fe0309adc54fb6a426bae02f26f898 with delete_type TABLET_DATA_TOMBSTONED (TS dede81a8c0164427a61532d01b6fc157 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:56490
I20250627 13:54:12.216317 11424 tablet_replica.cc:331] T 96fe0309adc54fb6a426bae02f26f898 P dede81a8c0164427a61532d01b6fc157: stopping tablet replica
I20250627 13:54:12.217952 11424 raft_consensus.cc:2241] T 96fe0309adc54fb6a426bae02f26f898 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:12.218590 7653 catalog_manager.cc:5582] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 reported cstate change: config changed from index 7 to 8, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:12.219376 11424 raft_consensus.cc:2270] T 96fe0309adc54fb6a426bae02f26f898 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:12.223421 7782 tablet_service.cc:1515] Processing DeleteTablet for tablet 14dba72d45694defb83f41c1572722ca with delete_type TABLET_DATA_TOMBSTONED (TS 83574c1bf8ea45adaff939158f93c096 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:45826
I20250627 13:54:12.223820 11389 tablet_replica.cc:331] T 14dba72d45694defb83f41c1572722ca P 83574c1bf8ea45adaff939158f93c096: stopping tablet replica
I20250627 13:54:12.224021 11389 raft_consensus.cc:2241] T 14dba72d45694defb83f41c1572722ca P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:12.224342 11389 raft_consensus.cc:2270] T 14dba72d45694defb83f41c1572722ca P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:12.225780 8193 consensus_queue.cc:237] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:12.227567 11389 ts_tablet_manager.cc:1905] T 14dba72d45694defb83f41c1572722ca P 83574c1bf8ea45adaff939158f93c096: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:12.234793 9840 raft_consensus.cc:1273] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:12.238199 10583 consensus_queue.cc:1035] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:12.239771 11389 ts_tablet_manager.cc:1918] T 14dba72d45694defb83f41c1572722ca P 83574c1bf8ea45adaff939158f93c096: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20250627 13:54:12.240027 11389 log.cc:1199] T 14dba72d45694defb83f41c1572722ca P 83574c1bf8ea45adaff939158f93c096: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/wals/14dba72d45694defb83f41c1572722ca
I20250627 13:54:12.241029 7645 catalog_manager.cc:4928] TS 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467): tablet 14dba72d45694defb83f41c1572722ca (table rebalance_test_table_6 [id=f83cab74056a4e7681a701169d1c1716]) successfully deleted
I20250627 13:54:12.243831 11051 raft_consensus.cc:2953] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:12.248117 7648 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet da90822ff15e4e5da05114ef38d3970a with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:12.250759 9840 raft_consensus.cc:2953] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:12.257593 7652 catalog_manager.cc:5582] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 7 to 8, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:12.263700 11335 raft_consensus.cc:2953] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:12.264474 7801 raft_consensus.cc:2953] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:12.265388 7931 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:12.266106 11034 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:12.267138 7782 tablet_service.cc:1515] Processing DeleteTablet for tablet da90822ff15e4e5da05114ef38d3970a with delete_type TABLET_DATA_TOMBSTONED (TS 83574c1bf8ea45adaff939158f93c096 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:45826
I20250627 13:54:12.268577 7931 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:12.270960 11389 tablet_replica.cc:331] T da90822ff15e4e5da05114ef38d3970a P 83574c1bf8ea45adaff939158f93c096: stopping tablet replica
I20250627 13:54:12.271231 11389 raft_consensus.cc:2241] T da90822ff15e4e5da05114ef38d3970a P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:12.272883 11389 raft_consensus.cc:2270] T da90822ff15e4e5da05114ef38d3970a P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:12.274101 11389 ts_tablet_manager.cc:1905] T da90822ff15e4e5da05114ef38d3970a P 83574c1bf8ea45adaff939158f93c096: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:12.278535 8063 raft_consensus.cc:2953] T 72db4bf98ab04b7189640f658d8fdcd7 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:12.280228 7926 raft_consensus.cc:1273] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:12.281229 11034 consensus_queue.cc:1035] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:12.282670 7926 raft_consensus.cc:2953] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:12.295589 11378 raft_consensus.cc:1062] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER 956ff058e6d64579a1963aabdb6d466b to VOTER
I20250627 13:54:12.296484 11378 consensus_queue.cc:237] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:12.298730 7659 catalog_manager.cc:5582] T 72db4bf98ab04b7189640f658d8fdcd7 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index -1 to 4, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:12.306185 9709 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:12.316622 7926 consensus_queue.cc:237] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:12.318982 11389 ts_tablet_manager.cc:1918] T da90822ff15e4e5da05114ef38d3970a P 83574c1bf8ea45adaff939158f93c096: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20250627 13:54:12.319157 11389 log.cc:1199] T da90822ff15e4e5da05114ef38d3970a P 83574c1bf8ea45adaff939158f93c096: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/wals/da90822ff15e4e5da05114ef38d3970a
I20250627 13:54:12.319923 7645 catalog_manager.cc:4928] TS 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467): tablet da90822ff15e4e5da05114ef38d3970a (table rebalance_test_table_2 [id=6defd4a16efc47569b2193b71e4b2c4a]) successfully deleted
I20250627 13:54:12.321758 8193 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:12.321910 9841 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:12.322312 11336 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:12.322919 11336 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:12.325168 11336 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:12.330416 9841 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:12.334734 7647 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 03640a5124d2405faf4a26d0e22e1ffc with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:12.337802 7800 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:12.337756 8193 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:12.341729 11188 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:12.342162 7653 catalog_manager.cc:5582] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index 7 to 8, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:12.343189 8190 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:12.344406 11382 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:12.344596 9711 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:12.346941 11336 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250627 13:54:12.348685 9711 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:12.352664 8041 tablet_service.cc:1515] Processing DeleteTablet for tablet 03640a5124d2405faf4a26d0e22e1ffc with delete_type TABLET_DATA_TOMBSTONED (TS 9cf479bdcd0242738d2a77db4ad9f660 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:59760
I20250627 13:54:12.361934 9841 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:12.363257 11382 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:12.365461 10084 tablet_service.cc:1515] Processing DeleteTablet for tablet da90822ff15e4e5da05114ef38d3970a with delete_type TABLET_DATA_TOMBSTONED (TS dede81a8c0164427a61532d01b6fc157 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56490
I20250627 13:54:12.367554 11424 ts_tablet_manager.cc:1905] T 96fe0309adc54fb6a426bae02f26f898 P dede81a8c0164427a61532d01b6fc157: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:12.385448 11436 tablet_replica.cc:331] T 03640a5124d2405faf4a26d0e22e1ffc P 9cf479bdcd0242738d2a77db4ad9f660: stopping tablet replica
I20250627 13:54:12.385708 11436 raft_consensus.cc:2241] T 03640a5124d2405faf4a26d0e22e1ffc P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:12.388477 11436 raft_consensus.cc:2270] T 03640a5124d2405faf4a26d0e22e1ffc P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:12.391223 11436 ts_tablet_manager.cc:1905] T 03640a5124d2405faf4a26d0e22e1ffc P 9cf479bdcd0242738d2a77db4ad9f660: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:12.389606 7926 consensus_queue.cc:237] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:12.396752 8193 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:12.396859 9841 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:12.397296 11336 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:12.397629 11021 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:12.400403 11335 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:12.405014 9842 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:12.407482 8193 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:12.412655 7647 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 03640a5124d2405faf4a26d0e22e1ffc with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:12.414300 7653 catalog_manager.cc:5582] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index 8 to 9, VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) evicted. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:12.417863 11188 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:12.424729 7654 catalog_manager.cc:5582] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 4 to 5, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:12.424836 8193 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:12.425535 11424 ts_tablet_manager.cc:1918] T 96fe0309adc54fb6a426bae02f26f898 P dede81a8c0164427a61532d01b6fc157: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20250627 13:54:12.425014 7801 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:12.426028 11424 log.cc:1199] T 96fe0309adc54fb6a426bae02f26f898 P dede81a8c0164427a61532d01b6fc157: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/wal/wals/96fe0309adc54fb6a426bae02f26f898
I20250627 13:54:12.432737 9842 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:12.434430 11436 ts_tablet_manager.cc:1918] T 03640a5124d2405faf4a26d0e22e1ffc P 9cf479bdcd0242738d2a77db4ad9f660: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20250627 13:54:12.454360 7647 catalog_manager.cc:4928] TS dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213): tablet 96fe0309adc54fb6a426bae02f26f898 (table rebalance_test_table_9 [id=19e5686b50fe4187a733aa6413ef2ce0]) successfully deleted
I20250627 13:54:12.443547 8063 consensus_queue.cc:237] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
W20250627 13:54:12.463493 7647 catalog_manager.cc:4908] TS dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213): delete failed for tablet da90822ff15e4e5da05114ef38d3970a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da90822ff15e4e5da05114ef38d3970a already in progress: copying tablet
I20250627 13:54:12.465955 7801 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:12.466552 9842 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:12.467121 11378 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:12.468235 11188 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:12.468958 11436 log.cc:1199] T 03640a5124d2405faf4a26d0e22e1ffc P 9cf479bdcd0242738d2a77db4ad9f660: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal/wals/03640a5124d2405faf4a26d0e22e1ffc
I20250627 13:54:12.470394 7645 catalog_manager.cc:4928] TS 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933): tablet 03640a5124d2405faf4a26d0e22e1ffc (table rebalance_test_table_1 [id=8961aa0df2f1488f82cd55cfa22ad214]) successfully deleted
I20250627 13:54:12.472460 11188 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:12.476012 9842 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:12.479879 7801 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:12.492338 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet e2bd060a4a9e4765b45b0242ae56e489 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:12.494995 7653 catalog_manager.cc:5582] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b reported cstate change: config changed from index 5 to 6, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:12.513849 8171 tablet_service.cc:1515] Processing DeleteTablet for tablet e2bd060a4a9e4765b45b0242ae56e489 with delete_type TABLET_DATA_TOMBSTONED (TS e4c831df4da841929d13d6c783099b0e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48616
I20250627 13:54:12.520133 7800 consensus_queue.cc:237] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:12.522701 7926 raft_consensus.cc:1273] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:12.523892 10928 consensus_queue.cc:1035] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:12.526139 9973 raft_consensus.cc:1273] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:12.526368 10961 tablet_bootstrap.cc:492] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:12.526798 10961 tablet_bootstrap.cc:492] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: Bootstrap complete.
I20250627 13:54:12.526831 10928 consensus_queue.cc:1035] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:12.527093 10961 ts_tablet_manager.cc:1397] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: Time spent bootstrapping tablet: real 0.595s user 0.054s sys 0.003s
I20250627 13:54:12.529764 10928 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:12.533327 10961 raft_consensus.cc:357] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:12.533625 10961 raft_consensus.cc:738] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 956ff058e6d64579a1963aabdb6d466b, State: Initialized, Role: LEARNER
I20250627 13:54:12.533944 10961 consensus_queue.cc:260] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:12.534817 10961 ts_tablet_manager.cc:1428] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20250627 13:54:12.535378 7926 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:12.537942 8213 tablet_copy_service.cc:342] P e4c831df4da841929d13d6c783099b0e: Request end of tablet copy session 956ff058e6d64579a1963aabdb6d466b-14dba72d45694defb83f41c1572722ca received from {username='slave'} at 127.0.103.70:33381
I20250627 13:54:12.538081 8213 tablet_copy_service.cc:434] P e4c831df4da841929d13d6c783099b0e: ending tablet copy session 956ff058e6d64579a1963aabdb6d466b-14dba72d45694defb83f41c1572722ca on tablet 14dba72d45694defb83f41c1572722ca with peer 956ff058e6d64579a1963aabdb6d466b
I20250627 13:54:12.539280 9822 tablet_service.cc:1515] Processing DeleteTablet for tablet 14dba72d45694defb83f41c1572722ca with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 6 (current committed config index is 8)) from {username='slave'} at 127.0.0.1:55228
I20250627 13:54:12.541186 9973 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:12.546659 7654 catalog_manager.cc:5582] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe reported cstate change: config changed from index 5 to 6, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:12.549996 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 40431e3bd7244904a4e65f59477bd21d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:12.550678 8172 tablet_service.cc:1515] Processing DeleteTablet for tablet 40431e3bd7244904a4e65f59477bd21d with delete_type TABLET_DATA_TOMBSTONED (TS e4c831df4da841929d13d6c783099b0e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48616
I20250627 13:54:12.570010 11273 raft_consensus.cc:1062] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096: attempting to promote NON_VOTER dede81a8c0164427a61532d01b6fc157 to VOTER
I20250627 13:54:12.571008 11273 consensus_queue.cc:237] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } }
I20250627 13:54:12.588498 11452 ts_tablet_manager.cc:927] T 72db4bf98ab04b7189640f658d8fdcd7 P 2a1282ebe92a4c9a92eed3b13f69012a: Initiating tablet copy from peer a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:54:12.590785 11451 tablet_replica.cc:331] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: stopping tablet replica
I20250627 13:54:12.591403 11451 raft_consensus.cc:2241] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Raft consensus shutting down.
I20250627 13:54:12.591835 11451 raft_consensus.cc:2270] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Raft consensus is shut down!
I20250627 13:54:12.593372 10103 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:12.594097 11274 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:12.603359 8193 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:12.604211 7931 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:12.605165 11272 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:12.606959 11272 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:12.608493 11449 tablet_replica.cc:331] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e: stopping tablet replica
I20250627 13:54:12.608815 11449 raft_consensus.cc:2241] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:12.608852 11452 tablet_copy_client.cc:323] T 72db4bf98ab04b7189640f658d8fdcd7 P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:43675
I20250627 13:54:12.609369 11449 raft_consensus.cc:2270] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:12.609473 7953 tablet_copy_service.cc:140] P a41a89ae8ed743edb2b5cf4490d8c854: Received BeginTabletCopySession request for tablet 72db4bf98ab04b7189640f658d8fdcd7 from peer 2a1282ebe92a4c9a92eed3b13f69012a ({username='slave'} at 127.0.103.69:49083)
I20250627 13:54:12.609688 7953 tablet_copy_service.cc:161] P a41a89ae8ed743edb2b5cf4490d8c854: Beginning new tablet copy session on tablet 72db4bf98ab04b7189640f658d8fdcd7 from peer 2a1282ebe92a4c9a92eed3b13f69012a at {username='slave'} at 127.0.103.69:49083: session id = 2a1282ebe92a4c9a92eed3b13f69012a-72db4bf98ab04b7189640f658d8fdcd7
I20250627 13:54:12.610677 11449 ts_tablet_manager.cc:1905] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:12.613602 7953 tablet_copy_source_session.cc:215] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:12.615217 11452 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72db4bf98ab04b7189640f658d8fdcd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:12.630939 10988 tablet_bootstrap.cc:492] T da90822ff15e4e5da05114ef38d3970a P dede81a8c0164427a61532d01b6fc157: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:12.631466 10988 tablet_bootstrap.cc:492] T da90822ff15e4e5da05114ef38d3970a P dede81a8c0164427a61532d01b6fc157: Bootstrap complete.
I20250627 13:54:12.631748 10988 ts_tablet_manager.cc:1397] T da90822ff15e4e5da05114ef38d3970a P dede81a8c0164427a61532d01b6fc157: Time spent bootstrapping tablet: real 0.576s user 0.047s sys 0.004s
I20250627 13:54:12.632669 10988 raft_consensus.cc:357] T da90822ff15e4e5da05114ef38d3970a P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:12.632956 10988 raft_consensus.cc:738] T da90822ff15e4e5da05114ef38d3970a P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dede81a8c0164427a61532d01b6fc157, State: Initialized, Role: LEARNER
I20250627 13:54:12.633205 10988 consensus_queue.cc:260] T da90822ff15e4e5da05114ef38d3970a P dede81a8c0164427a61532d01b6fc157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:12.633419 9691 tablet_service.cc:1515] Processing DeleteTablet for tablet 03640a5124d2405faf4a26d0e22e1ffc with delete_type TABLET_DATA_TOMBSTONED (TS 2a1282ebe92a4c9a92eed3b13f69012a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:53454
I20250627 13:54:12.633762 10988 ts_tablet_manager.cc:1428] T da90822ff15e4e5da05114ef38d3970a P dede81a8c0164427a61532d01b6fc157: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20250627 13:54:12.634441 8213 tablet_copy_service.cc:342] P e4c831df4da841929d13d6c783099b0e: Request end of tablet copy session dede81a8c0164427a61532d01b6fc157-da90822ff15e4e5da05114ef38d3970a received from {username='slave'} at 127.0.103.72:33995
I20250627 13:54:12.634578 8213 tablet_copy_service.cc:434] P e4c831df4da841929d13d6c783099b0e: ending tablet copy session dede81a8c0164427a61532d01b6fc157-da90822ff15e4e5da05114ef38d3970a on tablet da90822ff15e4e5da05114ef38d3970a with peer dede81a8c0164427a61532d01b6fc157
I20250627 13:54:12.636096 11452 tablet_copy_client.cc:806] T 72db4bf98ab04b7189640f658d8fdcd7 P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:12.636399 11452 tablet_copy_client.cc:670] T 72db4bf98ab04b7189640f658d8fdcd7 P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:12.638223 11452 tablet_copy_client.cc:538] T 72db4bf98ab04b7189640f658d8fdcd7 P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:12.643925 11274 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:12.646306 10084 tablet_service.cc:1515] Processing DeleteTablet for tablet da90822ff15e4e5da05114ef38d3970a with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 6 (current committed config index is 8)) from {username='slave'} at 127.0.0.1:56490
I20250627 13:54:12.646692 11424 tablet_replica.cc:331] T da90822ff15e4e5da05114ef38d3970a P dede81a8c0164427a61532d01b6fc157: stopping tablet replica
I20250627 13:54:12.646869 11424 raft_consensus.cc:2241] T da90822ff15e4e5da05114ef38d3970a P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Raft consensus shutting down.
I20250627 13:54:12.647058 11424 raft_consensus.cc:2270] T da90822ff15e4e5da05114ef38d3970a P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Raft consensus is shut down!
I20250627 13:54:12.647946 11424 ts_tablet_manager.cc:1905] T da90822ff15e4e5da05114ef38d3970a P dede81a8c0164427a61532d01b6fc157: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:12.649377 8193 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:12.651810 7653 catalog_manager.cc:5582] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 4 to 5, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:12.657318 11424 ts_tablet_manager.cc:1918] T da90822ff15e4e5da05114ef38d3970a P dede81a8c0164427a61532d01b6fc157: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20250627 13:54:12.662621 11424 log.cc:1199] T da90822ff15e4e5da05114ef38d3970a P dede81a8c0164427a61532d01b6fc157: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/wal/wals/da90822ff15e4e5da05114ef38d3970a
I20250627 13:54:12.662791 7800 consensus_queue.cc:237] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } }
I20250627 13:54:12.663522 7647 catalog_manager.cc:4928] TS dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213): tablet da90822ff15e4e5da05114ef38d3970a (table rebalance_test_table_2 [id=6defd4a16efc47569b2193b71e4b2c4a]) successfully deleted
I20250627 13:54:12.657122 10103 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:12.665771 8190 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:12.666321 11272 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:12.667542 11452 tablet_bootstrap.cc:492] T 72db4bf98ab04b7189640f658d8fdcd7 P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap starting.
I20250627 13:54:12.669761 11273 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:12.674727 11451 ts_tablet_manager.cc:1905] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:12.693107 10102 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:12.705616 11449 ts_tablet_manager.cc:1918] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250627 13:54:12.710263 11272 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:12.711769 11449 log.cc:1199] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal/wals/e2bd060a4a9e4765b45b0242ae56e489
I20250627 13:54:12.713289 10103 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:12.716516 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 897206e017ff4305a8cc9092e30f46b6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:12.718473 8193 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:12.720525 7653 catalog_manager.cc:5582] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 5 to 6, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:12.721159 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:12.721241 11449 tablet_replica.cc:331] T 40431e3bd7244904a4e65f59477bd21d P e4c831df4da841929d13d6c783099b0e: stopping tablet replica
I20250627 13:54:12.721894 7648 catalog_manager.cc:4928] TS e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): tablet e2bd060a4a9e4765b45b0242ae56e489 (table rebalance_test_table_6 [id=f83cab74056a4e7681a701169d1c1716]) successfully deleted
I20250627 13:54:12.723110 11466 tablet_replica.cc:331] T 03640a5124d2405faf4a26d0e22e1ffc P 2a1282ebe92a4c9a92eed3b13f69012a: stopping tablet replica
I20250627 13:54:12.725463 7911 tablet_service.cc:1515] Processing DeleteTablet for tablet 897206e017ff4305a8cc9092e30f46b6 with delete_type TABLET_DATA_TOMBSTONED (TS a41a89ae8ed743edb2b5cf4490d8c854 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59302
I20250627 13:54:12.741925 11378 raft_consensus.cc:1062] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER 6686c6c365bf4f4582058890161676fe to VOTER
I20250627 13:54:12.742790 11378 consensus_queue.cc:237] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:12.743456 11466 raft_consensus.cc:2241] T 03640a5124d2405faf4a26d0e22e1ffc P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:12.743968 11466 raft_consensus.cc:2270] T 03640a5124d2405faf4a26d0e22e1ffc P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:12.760392 11449 raft_consensus.cc:2241] T 40431e3bd7244904a4e65f59477bd21d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:12.761202 11449 raft_consensus.cc:2270] T 40431e3bd7244904a4e65f59477bd21d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:12.765681 11449 ts_tablet_manager.cc:1905] T 40431e3bd7244904a4e65f59477bd21d P e4c831df4da841929d13d6c783099b0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:12.773293 11475 tablet_replica.cc:331] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854: stopping tablet replica
I20250627 13:54:12.774653 11475 raft_consensus.cc:2241] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:12.775566 11475 pending_rounds.cc:70] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854: Trying to abort 1 pending ops.
I20250627 13:54:12.776124 11475 pending_rounds.cc:77] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7172228925728129024 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "897206e017ff4305a8cc9092e30f46b6" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } } }
I20250627 13:54:12.779037 11475 raft_consensus.cc:2915] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20250627 13:54:12.779699 11475 raft_consensus.cc:2887] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250627 13:54:12.780925 11475 raft_consensus.cc:2270] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:12.781279 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:12.783392 11475 ts_tablet_manager.cc:1905] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:12.786211 11466 ts_tablet_manager.cc:1905] T 03640a5124d2405faf4a26d0e22e1ffc P 2a1282ebe92a4c9a92eed3b13f69012a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:12.793128 8190 raft_consensus.cc:1273] T 5184c3f8e465472e9a98c14f8e703415 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:12.793150 9973 raft_consensus.cc:1273] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:12.793697 11382 consensus_queue.cc:1035] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:12.795012 7800 raft_consensus.cc:1273] T 5184c3f8e465472e9a98c14f8e703415 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:12.795480 11378 consensus_queue.cc:1035] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:12.795785 11382 consensus_queue.cc:1035] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:12.813839 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:12.826956 11475 ts_tablet_manager.cc:1918] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250627 13:54:12.829541 11475 log.cc:1199] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/wals/897206e017ff4305a8cc9092e30f46b6
I20250627 13:54:12.830116 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:12.831372 7647 catalog_manager.cc:4928] TS a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675): tablet 897206e017ff4305a8cc9092e30f46b6 (table rebalance_test_table_4 [id=4d92d465970b4ec4b5eace263b25362a]) successfully deleted
I20250627 13:54:12.837471 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:12.891494 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:12.895464 11188 raft_consensus.cc:2953] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:12.903232 7659 catalog_manager.cc:5582] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:12.904603 8189 raft_consensus.cc:2953] T 5184c3f8e465472e9a98c14f8e703415 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:12.916285 9973 raft_consensus.cc:2953] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:12.923848 11449 ts_tablet_manager.cc:1918] T 40431e3bd7244904a4e65f59477bd21d P e4c831df4da841929d13d6c783099b0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250627 13:54:12.924144 11449 log.cc:1199] T 40431e3bd7244904a4e65f59477bd21d P e4c831df4da841929d13d6c783099b0e: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal/wals/40431e3bd7244904a4e65f59477bd21d
I20250627 13:54:12.924934 8063 consensus_queue.cc:237] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:12.925920 7648 catalog_manager.cc:4928] TS e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): tablet 40431e3bd7244904a4e65f59477bd21d (table rebalance_test_table_2 [id=6defd4a16efc47569b2193b71e4b2c4a]) successfully deleted
I20250627 13:54:12.927181 9973 raft_consensus.cc:1273] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:12.927695 11188 consensus_queue.cc:1035] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:12.928248 7801 raft_consensus.cc:1273] T 5184c3f8e465472e9a98c14f8e703415 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:12.930305 11382 consensus_queue.cc:1035] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:12.931268 11382 raft_consensus.cc:2953] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:12.931375 7801 raft_consensus.cc:2953] T 5184c3f8e465472e9a98c14f8e703415 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:12.933636 9972 raft_consensus.cc:2953] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:12.937255 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 5184c3f8e465472e9a98c14f8e703415 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:12.941947 7654 catalog_manager.cc:5582] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe reported cstate change: config changed from index 5 to 6, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:12.945574 8171 tablet_service.cc:1515] Processing DeleteTablet for tablet 5184c3f8e465472e9a98c14f8e703415 with delete_type TABLET_DATA_TOMBSTONED (TS e4c831df4da841929d13d6c783099b0e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48616
I20250627 13:54:12.947206 11449 tablet_replica.cc:331] T 5184c3f8e465472e9a98c14f8e703415 P e4c831df4da841929d13d6c783099b0e: stopping tablet replica
I20250627 13:54:12.947248 10084 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:12.947597 11449 raft_consensus.cc:2241] T 5184c3f8e465472e9a98c14f8e703415 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:12.947968 11449 raft_consensus.cc:2270] T 5184c3f8e465472e9a98c14f8e703415 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:12.948848 7800 raft_consensus.cc:2953] T 5184c3f8e465472e9a98c14f8e703415 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:12.952797 11449 ts_tablet_manager.cc:1905] T 5184c3f8e465472e9a98c14f8e703415 P e4c831df4da841929d13d6c783099b0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:12.973269 11449 ts_tablet_manager.cc:1918] T 5184c3f8e465472e9a98c14f8e703415 P e4c831df4da841929d13d6c783099b0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250627 13:54:12.987906 11449 log.cc:1199] T 5184c3f8e465472e9a98c14f8e703415 P e4c831df4da841929d13d6c783099b0e: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal/wals/5184c3f8e465472e9a98c14f8e703415
I20250627 13:54:12.990224 7648 catalog_manager.cc:4928] TS e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): tablet 5184c3f8e465472e9a98c14f8e703415 (table rebalance_test_table_3 [id=ec6a9ed6ab4d416988b2abe2f8d29c55]) successfully deleted
I20250627 13:54:12.999518 11466 ts_tablet_manager.cc:1918] T 03640a5124d2405faf4a26d0e22e1ffc P 2a1282ebe92a4c9a92eed3b13f69012a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20250627 13:54:12.999779 11466 log.cc:1199] T 03640a5124d2405faf4a26d0e22e1ffc P 2a1282ebe92a4c9a92eed3b13f69012a: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal/wals/03640a5124d2405faf4a26d0e22e1ffc
I20250627 13:54:13.000801 7648 catalog_manager.cc:4928] TS 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101): tablet 03640a5124d2405faf4a26d0e22e1ffc (table rebalance_test_table_1 [id=8961aa0df2f1488f82cd55cfa22ad214]) successfully deleted
I20250627 13:54:13.001735 11451 ts_tablet_manager.cc:1918] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20250627 13:54:13.002527 11451 log.cc:1199] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal/wals/14dba72d45694defb83f41c1572722ca
I20250627 13:54:13.003242 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:13.003502 7645 catalog_manager.cc:4928] TS 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): tablet 14dba72d45694defb83f41c1572722ca (table rebalance_test_table_6 [id=f83cab74056a4e7681a701169d1c1716]) successfully deleted
I20250627 13:54:13.043289 8063 consensus_queue.cc:237] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
W20250627 13:54:13.078872 7996 consensus_peers.cc:487] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9196497cb11f4ccbbd62695253d4c8c9. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:13.079278 8189 raft_consensus.cc:1273] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:13.079854 7800 raft_consensus.cc:1273] T 9196497cb11f4ccbbd62695253d4c8c9 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:13.080581 11382 consensus_queue.cc:1035] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:13.082433 11188 consensus_queue.cc:1035] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:13.092406 11382 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:13.093956 8193 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:13.097043 7800 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:13.099309 7659 catalog_manager.cc:5582] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:13.190323 11452 tablet_bootstrap.cc:492] T 72db4bf98ab04b7189640f658d8fdcd7 P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:13.190879 11452 tablet_bootstrap.cc:492] T 72db4bf98ab04b7189640f658d8fdcd7 P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap complete.
I20250627 13:54:13.191231 11452 ts_tablet_manager.cc:1397] T 72db4bf98ab04b7189640f658d8fdcd7 P 2a1282ebe92a4c9a92eed3b13f69012a: Time spent bootstrapping tablet: real 0.524s user 0.057s sys 0.000s
I20250627 13:54:13.192301 11452 raft_consensus.cc:357] T 72db4bf98ab04b7189640f658d8fdcd7 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:13.192586 11452 raft_consensus.cc:738] T 72db4bf98ab04b7189640f658d8fdcd7 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a1282ebe92a4c9a92eed3b13f69012a, State: Initialized, Role: LEARNER
I20250627 13:54:13.192816 11452 consensus_queue.cc:260] T 72db4bf98ab04b7189640f658d8fdcd7 P 2a1282ebe92a4c9a92eed3b13f69012a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:13.222905 11452 ts_tablet_manager.cc:1428] T 72db4bf98ab04b7189640f658d8fdcd7 P 2a1282ebe92a4c9a92eed3b13f69012a: Time spent starting tablet: real 0.032s user 0.003s sys 0.000s
I20250627 13:54:13.224088 7953 tablet_copy_service.cc:342] P a41a89ae8ed743edb2b5cf4490d8c854: Request end of tablet copy session 2a1282ebe92a4c9a92eed3b13f69012a-72db4bf98ab04b7189640f658d8fdcd7 received from {username='slave'} at 127.0.103.69:49083
I20250627 13:54:13.224224 7953 tablet_copy_service.cc:434] P a41a89ae8ed743edb2b5cf4490d8c854: ending tablet copy session 2a1282ebe92a4c9a92eed3b13f69012a-72db4bf98ab04b7189640f658d8fdcd7 on tablet 72db4bf98ab04b7189640f658d8fdcd7 with peer 2a1282ebe92a4c9a92eed3b13f69012a
I20250627 13:54:13.241158 7931 consensus_queue.cc:237] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:13.246414 8189 consensus_queue.cc:237] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
W20250627 13:54:13.272840 7866 consensus_peers.cc:487] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cff879c8e57d42c28440b3bd310f1ba9. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:13.274664 7800 raft_consensus.cc:1273] T cff879c8e57d42c28440b3bd310f1ba9 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:13.277336 11336 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:13.284971 8063 raft_consensus.cc:1273] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:13.285646 11336 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:13.300118 7931 raft_consensus.cc:1273] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:13.300801 11493 consensus_queue.cc:1035] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:13.301012 8063 raft_consensus.cc:1273] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:13.301631 11493 consensus_queue.cc:1035] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:13.310307 8062 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:13.309363 11336 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:13.317620 7659 catalog_manager.cc:5582] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:13.324699 11493 raft_consensus.cc:2953] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:13.328482 7926 raft_consensus.cc:2953] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:13.335601 7659 catalog_manager.cc:5582] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index -1 to 4, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:13.337865 8062 raft_consensus.cc:2953] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:13.367152 7801 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
W20250627 13:54:13.376775 8129 consensus_peers.cc:487] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e -> Peer 6686c6c365bf4f4582058890161676fe (127.0.103.71:37859): Couldn't send request to peer 6686c6c365bf4f4582058890161676fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ab02f13e45c48d293e2d2d229d11e27. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:13.410334 8189 consensus_queue.cc:237] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
W20250627 13:54:13.425953 8129 consensus_peers.cc:487] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e -> Peer 6686c6c365bf4f4582058890161676fe (127.0.103.71:37859): Couldn't send request to peer 6686c6c365bf4f4582058890161676fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e3053b529a264e8697a1df9e17eeea81. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:13.427318 9709 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:13.427946 11493 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:13.435508 8062 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:13.436220 11493 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:13.451970 11493 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:13.456346 8061 consensus_queue.cc:237] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } }
I20250627 13:54:13.459882 9709 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:13.468189 7800 consensus_queue.cc:237] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:13.471700 7654 catalog_manager.cc:5582] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: config changed from index 6 to 7, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:13.493492 10103 raft_consensus.cc:1273] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:13.494354 11497 consensus_queue.cc:1035] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:13.510290 7801 raft_consensus.cc:1273] T 91664403eed74badbeba83d69f597e1f P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:13.511062 11497 consensus_queue.cc:1035] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:13.512075 7931 raft_consensus.cc:1273] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:13.512679 11382 consensus_queue.cc:1035] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:13.513773 8061 raft_consensus.cc:1273] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:13.514879 7931 raft_consensus.cc:1273] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:13.515409 11273 consensus_queue.cc:1035] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:13.516793 11274 consensus_queue.cc:1035] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:13.525282 9711 raft_consensus.cc:1215] T 72db4bf98ab04b7189640f658d8fdcd7 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250627 13:54:13.531011 11274 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:13.534312 7933 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:13.541482 7654 catalog_manager.cc:5582] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index -1 to 4, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:13.546620 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:13.548874 8061 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
W20250627 13:54:13.553426 7738 consensus_peers.cc:487] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 -> Peer 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101): Couldn't send request to peer 2a1282ebe92a4c9a92eed3b13f69012a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1dc569a7b12d40058d073f7b81b2601c. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:13.556686 8063 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:13.581559 11382 raft_consensus.cc:2953] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:13.585994 7801 raft_consensus.cc:2953] T 91664403eed74badbeba83d69f597e1f P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:13.588042 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:13.591794 7655 catalog_manager.cc:5582] T 91664403eed74badbeba83d69f597e1f P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 5 to 6. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:13.597144 10103 raft_consensus.cc:2953] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:13.601359 8061 consensus_queue.cc:237] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
W20250627 13:54:13.606093 7999 consensus_peers.cc:487] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 6686c6c365bf4f4582058890161676fe (127.0.103.71:37859): Couldn't send request to peer 6686c6c365bf4f4582058890161676fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91664403eed74badbeba83d69f597e1f. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:13.607511 7801 raft_consensus.cc:1273] T 91664403eed74badbeba83d69f597e1f P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:13.608176 11188 consensus_queue.cc:1035] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:13.610852 7933 raft_consensus.cc:1273] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:13.611694 11382 consensus_queue.cc:1035] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:13.612932 7933 raft_consensus.cc:2953] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:13.624256 11188 raft_consensus.cc:2953] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:13.629523 7645 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 91664403eed74badbeba83d69f597e1f with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250627 13:54:13.632181 7801 raft_consensus.cc:2953] T 91664403eed74badbeba83d69f597e1f P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:13.632725 7655 catalog_manager.cc:5582] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 6 to 7, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:13.643275 7926 raft_consensus.cc:2953] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:13.648557 10102 raft_consensus.cc:1273] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:13.649698 11382 consensus_queue.cc:1035] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:13.651476 10102 raft_consensus.cc:2953] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:13.679523 11607 ts_tablet_manager.cc:927] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:13.680946 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:13.690105 11607 tablet_copy_client.cc:323] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:13.690982 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet 9196497cb11f4ccbbd62695253d4c8c9 from peer 956ff058e6d64579a1963aabdb6d466b ({username='slave'} at 127.0.103.70:59981)
I20250627 13:54:13.691177 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet 9196497cb11f4ccbbd62695253d4c8c9 from peer 956ff058e6d64579a1963aabdb6d466b at {username='slave'} at 127.0.103.70:59981: session id = 956ff058e6d64579a1963aabdb6d466b-9196497cb11f4ccbbd62695253d4c8c9
I20250627 13:54:13.693589 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:13.695254 8082 tablet_copy_source_session.cc:215] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:13.696936 11607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9196497cb11f4ccbbd62695253d4c8c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:13.706578 11607 tablet_copy_client.cc:806] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:13.706976 11607 tablet_copy_client.cc:670] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:13.708822 11607 tablet_copy_client.cc:538] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:13.709177 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:13.713063 11607 tablet_bootstrap.cc:492] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap starting.
I20250627 13:54:13.776814 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:13.783248 10084 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:13.827687 11624 ts_tablet_manager.cc:927] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b: Initiating tablet copy from peer a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:54:13.839058 11624 tablet_copy_client.cc:323] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:43675
I20250627 13:54:13.839996 7953 tablet_copy_service.cc:140] P a41a89ae8ed743edb2b5cf4490d8c854: Received BeginTabletCopySession request for tablet cff879c8e57d42c28440b3bd310f1ba9 from peer 956ff058e6d64579a1963aabdb6d466b ({username='slave'} at 127.0.103.70:60171)
I20250627 13:54:13.840163 7953 tablet_copy_service.cc:161] P a41a89ae8ed743edb2b5cf4490d8c854: Beginning new tablet copy session on tablet cff879c8e57d42c28440b3bd310f1ba9 from peer 956ff058e6d64579a1963aabdb6d466b at {username='slave'} at 127.0.103.70:60171: session id = 956ff058e6d64579a1963aabdb6d466b-cff879c8e57d42c28440b3bd310f1ba9
I20250627 13:54:13.843361 7953 tablet_copy_source_session.cc:215] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:13.845423 11607 tablet_bootstrap.cc:492] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:13.850440 11624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cff879c8e57d42c28440b3bd310f1ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:13.853560 11607 tablet_bootstrap.cc:492] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap complete.
I20250627 13:54:13.854385 11607 ts_tablet_manager.cc:1397] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b: Time spent bootstrapping tablet: real 0.141s user 0.054s sys 0.006s
I20250627 13:54:13.855324 11607 raft_consensus.cc:357] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:13.857067 11607 raft_consensus.cc:738] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 956ff058e6d64579a1963aabdb6d466b, State: Initialized, Role: LEARNER
I20250627 13:54:13.857383 11607 consensus_queue.cc:260] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:13.867013 8189 consensus_queue.cc:237] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:13.869966 7801 raft_consensus.cc:1273] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:13.870527 8061 raft_consensus.cc:1273] T 5f176b66fdce4398be3492a5f3b4ea27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:13.871320 11493 consensus_queue.cc:1035] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250627 13:54:13.871732 8128 consensus_peers.cc:487] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e -> Peer dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213): Couldn't send request to peer dede81a8c0164427a61532d01b6fc157. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f176b66fdce4398be3492a5f3b4ea27. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:13.872326 11492 consensus_queue.cc:1035] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:13.883466 11492 raft_consensus.cc:2953] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:13.887534 7800 raft_consensus.cc:2953] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:13.892768 7655 catalog_manager.cc:5582] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:13.903746 11555 raft_consensus.cc:1062] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854: attempting to promote NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a to VOTER
I20250627 13:54:13.904740 11555 consensus_queue.cc:237] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:13.916376 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:13.940652 9709 raft_consensus.cc:1273] T 72db4bf98ab04b7189640f658d8fdcd7 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:13.941334 11021 consensus_queue.cc:1035] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:13.949661 8062 raft_consensus.cc:1273] T 72db4bf98ab04b7189640f658d8fdcd7 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:13.950028 7800 raft_consensus.cc:1273] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:13.950850 11336 consensus_queue.cc:1035] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:13.950821 8061 raft_consensus.cc:2953] T 5f176b66fdce4398be3492a5f3b4ea27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:13.951572 11336 consensus_queue.cc:1035] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:13.957307 11635 ts_tablet_manager.cc:927] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe: Initiating tablet copy from peer e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:54:13.962005 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:13.971251 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:13.973497 11021 raft_consensus.cc:2953] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:13.982275 8062 raft_consensus.cc:2953] T 72db4bf98ab04b7189640f658d8fdcd7 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:13.982429 7800 raft_consensus.cc:2953] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:13.989327 7655 catalog_manager.cc:5582] T 72db4bf98ab04b7189640f658d8fdcd7 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:13.996958 9711 raft_consensus.cc:2953] T 72db4bf98ab04b7189640f658d8fdcd7 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:13.999382 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:14.010241 7926 consensus_queue.cc:237] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:14.010882 11624 tablet_copy_client.cc:806] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:14.011201 11624 tablet_copy_client.cc:670] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:14.013867 9709 raft_consensus.cc:1273] T 72db4bf98ab04b7189640f658d8fdcd7 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:14.013957 7800 raft_consensus.cc:1273] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:14.014434 11555 consensus_queue.cc:1035] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:14.014725 11021 consensus_queue.cc:1035] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:14.018707 11607 ts_tablet_manager.cc:1428] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b: Time spent starting tablet: real 0.164s user 0.039s sys 0.070s
I20250627 13:54:14.019883 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session 956ff058e6d64579a1963aabdb6d466b-9196497cb11f4ccbbd62695253d4c8c9 received from {username='slave'} at 127.0.103.70:59981
I20250627 13:54:14.020008 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session 956ff058e6d64579a1963aabdb6d466b-9196497cb11f4ccbbd62695253d4c8c9 on tablet 9196497cb11f4ccbbd62695253d4c8c9 with peer 956ff058e6d64579a1963aabdb6d466b
I20250627 13:54:14.019814 11021 raft_consensus.cc:2953] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:14.027757 7647 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 72db4bf98ab04b7189640f658d8fdcd7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:14.029119 11624 tablet_copy_client.cc:538] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:14.028875 9709 raft_consensus.cc:2953] T 72db4bf98ab04b7189640f658d8fdcd7 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:14.030287 7655 catalog_manager.cc:5582] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index 5 to 6, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:14.030344 7800 raft_consensus.cc:2953] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:14.033921 8041 tablet_service.cc:1515] Processing DeleteTablet for tablet 72db4bf98ab04b7189640f658d8fdcd7 with delete_type TABLET_DATA_TOMBSTONED (TS 9cf479bdcd0242738d2a77db4ad9f660 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59760
I20250627 13:54:14.056169 11624 tablet_bootstrap.cc:492] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap starting.
I20250627 13:54:14.062146 11635 tablet_copy_client.cc:323] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.68:46139
I20250627 13:54:14.067657 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:14.086583 11645 tablet_replica.cc:331] T 72db4bf98ab04b7189640f658d8fdcd7 P 9cf479bdcd0242738d2a77db4ad9f660: stopping tablet replica
I20250627 13:54:14.086871 11645 raft_consensus.cc:2241] T 72db4bf98ab04b7189640f658d8fdcd7 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:14.087119 11645 raft_consensus.cc:2270] T 72db4bf98ab04b7189640f658d8fdcd7 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:14.088070 11645 ts_tablet_manager.cc:1905] T 72db4bf98ab04b7189640f658d8fdcd7 P 9cf479bdcd0242738d2a77db4ad9f660: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:14.094933 11645 ts_tablet_manager.cc:1918] T 72db4bf98ab04b7189640f658d8fdcd7 P 9cf479bdcd0242738d2a77db4ad9f660: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250627 13:54:14.095131 11645 log.cc:1199] T 72db4bf98ab04b7189640f658d8fdcd7 P 9cf479bdcd0242738d2a77db4ad9f660: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal/wals/72db4bf98ab04b7189640f658d8fdcd7
I20250627 13:54:14.096167 7645 catalog_manager.cc:4928] TS 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933): tablet 72db4bf98ab04b7189640f658d8fdcd7 (table rebalance_test_table_8 [id=b84f3fe3fd7c4f01902547d2db982f90]) successfully deleted
I20250627 13:54:14.097312 11636 ts_tablet_manager.cc:927] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe: Initiating tablet copy from peer e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:54:14.098239 11647 ts_tablet_manager.cc:927] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a: Initiating tablet copy from peer 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:54:14.102046 11636 tablet_copy_client.cc:323] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.68:46139
I20250627 13:54:14.134114 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:14.135527 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:14.136437 8062 consensus_queue.cc:237] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:14.140026 10103 raft_consensus.cc:1273] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:14.141391 11382 consensus_queue.cc:1035] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:14.143159 7926 raft_consensus.cc:1273] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:14.143707 11382 consensus_queue.cc:1035] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20250627 13:54:14.145673 7999 consensus_peers.cc:487] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 6686c6c365bf4f4582058890161676fe (127.0.103.71:37859): Couldn't send request to peer 6686c6c365bf4f4582058890161676fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91664403eed74badbeba83d69f597e1f. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:14.146047 11647 tablet_copy_client.cc:323] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:37467
I20250627 13:54:14.146821 7822 tablet_copy_service.cc:140] P 83574c1bf8ea45adaff939158f93c096: Received BeginTabletCopySession request for tablet 1dc569a7b12d40058d073f7b81b2601c from peer 2a1282ebe92a4c9a92eed3b13f69012a ({username='slave'} at 127.0.103.69:43835)
I20250627 13:54:14.146996 7822 tablet_copy_service.cc:161] P 83574c1bf8ea45adaff939158f93c096: Beginning new tablet copy session on tablet 1dc569a7b12d40058d073f7b81b2601c from peer 2a1282ebe92a4c9a92eed3b13f69012a at {username='slave'} at 127.0.103.69:43835: session id = 2a1282ebe92a4c9a92eed3b13f69012a-1dc569a7b12d40058d073f7b81b2601c
I20250627 13:54:14.147774 11378 raft_consensus.cc:2953] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:14.150269 7822 tablet_copy_source_session.cc:215] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:14.152066 11647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dc569a7b12d40058d073f7b81b2601c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:14.152508 10103 raft_consensus.cc:2953] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:14.157388 8213 tablet_copy_service.cc:140] P e4c831df4da841929d13d6c783099b0e: Received BeginTabletCopySession request for tablet 8ab02f13e45c48d293e2d2d229d11e27 from peer 6686c6c365bf4f4582058890161676fe ({username='slave'} at 127.0.103.71:35925)
I20250627 13:54:14.157593 8213 tablet_copy_service.cc:161] P e4c831df4da841929d13d6c783099b0e: Beginning new tablet copy session on tablet 8ab02f13e45c48d293e2d2d229d11e27 from peer 6686c6c365bf4f4582058890161676fe at {username='slave'} at 127.0.103.71:35925: session id = 6686c6c365bf4f4582058890161676fe-8ab02f13e45c48d293e2d2d229d11e27
I20250627 13:54:14.160818 8213 tablet_copy_source_session.cc:215] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:14.161219 7926 raft_consensus.cc:2953] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:14.162135 8213 tablet_copy_service.cc:140] P e4c831df4da841929d13d6c783099b0e: Received BeginTabletCopySession request for tablet e3053b529a264e8697a1df9e17eeea81 from peer 6686c6c365bf4f4582058890161676fe ({username='slave'} at 127.0.103.71:35925)
I20250627 13:54:14.162292 8213 tablet_copy_service.cc:161] P e4c831df4da841929d13d6c783099b0e: Beginning new tablet copy session on tablet e3053b529a264e8697a1df9e17eeea81 from peer 6686c6c365bf4f4582058890161676fe at {username='slave'} at 127.0.103.71:35925: session id = 6686c6c365bf4f4582058890161676fe-e3053b529a264e8697a1df9e17eeea81
I20250627 13:54:14.165858 8213 tablet_copy_source_session.cc:215] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:14.167090 11635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ab02f13e45c48d293e2d2d229d11e27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:14.168845 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 91664403eed74badbeba83d69f597e1f with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:14.179280 7655 catalog_manager.cc:5582] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157 reported cstate change: config changed from index 7 to 8, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:14.183868 11647 tablet_copy_client.cc:806] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:14.184438 11647 tablet_copy_client.cc:670] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:14.187664 7782 tablet_service.cc:1515] Processing DeleteTablet for tablet 91664403eed74badbeba83d69f597e1f with delete_type TABLET_DATA_TOMBSTONED (TS 83574c1bf8ea45adaff939158f93c096 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:45826
I20250627 13:54:14.207516 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:14.208510 9842 raft_consensus.cc:1215] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250627 13:54:14.215093 11663 tablet_replica.cc:331] T 91664403eed74badbeba83d69f597e1f P 83574c1bf8ea45adaff939158f93c096: stopping tablet replica
I20250627 13:54:14.215411 11663 raft_consensus.cc:2241] T 91664403eed74badbeba83d69f597e1f P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:14.216832 11636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3053b529a264e8697a1df9e17eeea81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:14.216142 11663 raft_consensus.cc:2270] T 91664403eed74badbeba83d69f597e1f P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:14.241355 11636 tablet_copy_client.cc:806] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:14.241624 11636 tablet_copy_client.cc:670] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:14.243476 10084 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:14.261018 11647 tablet_copy_client.cc:538] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:14.265357 11636 tablet_copy_client.cc:538] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:14.267306 8061 consensus_queue.cc:237] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:14.269381 11636 tablet_bootstrap.cc:492] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe: Bootstrap starting.
I20250627 13:54:14.278317 11624 tablet_bootstrap.cc:492] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:14.278761 11624 tablet_bootstrap.cc:492] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap complete.
I20250627 13:54:14.279047 11624 ts_tablet_manager.cc:1397] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b: Time spent bootstrapping tablet: real 0.223s user 0.043s sys 0.017s
I20250627 13:54:14.279841 11624 raft_consensus.cc:357] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:14.280069 11624 raft_consensus.cc:738] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 956ff058e6d64579a1963aabdb6d466b, State: Initialized, Role: LEARNER
I20250627 13:54:14.280326 11624 consensus_queue.cc:260] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:14.281702 11624 ts_tablet_manager.cc:1428] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250627 13:54:14.282433 7953 tablet_copy_service.cc:342] P a41a89ae8ed743edb2b5cf4490d8c854: Request end of tablet copy session 956ff058e6d64579a1963aabdb6d466b-cff879c8e57d42c28440b3bd310f1ba9 received from {username='slave'} at 127.0.103.70:60171
I20250627 13:54:14.282605 7953 tablet_copy_service.cc:434] P a41a89ae8ed743edb2b5cf4490d8c854: ending tablet copy session 956ff058e6d64579a1963aabdb6d466b-cff879c8e57d42c28440b3bd310f1ba9 on tablet cff879c8e57d42c28440b3bd310f1ba9 with peer 956ff058e6d64579a1963aabdb6d466b
I20250627 13:54:14.286635 11663 ts_tablet_manager.cc:1905] T 91664403eed74badbeba83d69f597e1f P 83574c1bf8ea45adaff939158f93c096: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:14.289456 11635 tablet_copy_client.cc:806] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:14.289820 11635 tablet_copy_client.cc:670] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:14.292944 8189 raft_consensus.cc:1273] T 00feb28e9047435dbe36a595b1e23fdd P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:14.293004 11635 tablet_copy_client.cc:538] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:14.293561 11378 consensus_queue.cc:1035] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250627 13:54:14.296124 7996 consensus_peers.cc:487] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00feb28e9047435dbe36a595b1e23fdd. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:14.296953 11635 tablet_bootstrap.cc:492] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe: Bootstrap starting.
I20250627 13:54:14.299693 11647 tablet_bootstrap.cc:492] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap starting.
I20250627 13:54:14.300344 7800 raft_consensus.cc:1273] T 00feb28e9047435dbe36a595b1e23fdd P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:14.301002 11378 consensus_queue.cc:1035] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:14.304555 11188 raft_consensus.cc:2953] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:14.310899 7800 raft_consensus.cc:2953] T 00feb28e9047435dbe36a595b1e23fdd P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:14.317492 7655 catalog_manager.cc:5582] T 00feb28e9047435dbe36a595b1e23fdd P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:14.342283 9842 raft_consensus.cc:1215] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250627 13:54:14.358307 11663 ts_tablet_manager.cc:1918] T 91664403eed74badbeba83d69f597e1f P 83574c1bf8ea45adaff939158f93c096: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20250627 13:54:14.370111 11663 log.cc:1199] T 91664403eed74badbeba83d69f597e1f P 83574c1bf8ea45adaff939158f93c096: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/wals/91664403eed74badbeba83d69f597e1f
I20250627 13:54:14.371356 7645 catalog_manager.cc:4928] TS 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467): tablet 91664403eed74badbeba83d69f597e1f (table rebalance_test_table_7 [id=46b91111142d4e388d01cd06d3d41940]) successfully deleted
I20250627 13:54:14.374687 8193 raft_consensus.cc:2953] T 00feb28e9047435dbe36a595b1e23fdd P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:14.440107 11679 ts_tablet_manager.cc:927] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157: Initiating tablet copy from peer e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:54:14.452381 11679 tablet_copy_client.cc:323] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.68:46139
I20250627 13:54:14.457580 8213 tablet_copy_service.cc:140] P e4c831df4da841929d13d6c783099b0e: Received BeginTabletCopySession request for tablet 5f176b66fdce4398be3492a5f3b4ea27 from peer dede81a8c0164427a61532d01b6fc157 ({username='slave'} at 127.0.103.72:33995)
I20250627 13:54:14.457818 8213 tablet_copy_service.cc:161] P e4c831df4da841929d13d6c783099b0e: Beginning new tablet copy session on tablet 5f176b66fdce4398be3492a5f3b4ea27 from peer dede81a8c0164427a61532d01b6fc157 at {username='slave'} at 127.0.103.72:33995: session id = dede81a8c0164427a61532d01b6fc157-5f176b66fdce4398be3492a5f3b4ea27
I20250627 13:54:14.460839 8213 tablet_copy_source_session.cc:215] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:14.465663 11679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f176b66fdce4398be3492a5f3b4ea27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:14.468956 11647 tablet_bootstrap.cc:492] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:14.469727 11647 tablet_bootstrap.cc:492] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap complete.
I20250627 13:54:14.470049 11647 ts_tablet_manager.cc:1397] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a: Time spent bootstrapping tablet: real 0.170s user 0.044s sys 0.012s
I20250627 13:54:14.470757 11647 raft_consensus.cc:357] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:14.470978 11647 raft_consensus.cc:738] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a1282ebe92a4c9a92eed3b13f69012a, State: Initialized, Role: LEARNER
I20250627 13:54:14.471210 11647 consensus_queue.cc:260] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:14.472844 11647 ts_tablet_manager.cc:1428] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250627 13:54:14.473642 7822 tablet_copy_service.cc:342] P 83574c1bf8ea45adaff939158f93c096: Request end of tablet copy session 2a1282ebe92a4c9a92eed3b13f69012a-1dc569a7b12d40058d073f7b81b2601c received from {username='slave'} at 127.0.103.69:43835
I20250627 13:54:14.473825 7822 tablet_copy_service.cc:434] P 83574c1bf8ea45adaff939158f93c096: ending tablet copy session 2a1282ebe92a4c9a92eed3b13f69012a-1dc569a7b12d40058d073f7b81b2601c on tablet 1dc569a7b12d40058d073f7b81b2601c with peer 2a1282ebe92a4c9a92eed3b13f69012a
I20250627 13:54:14.478569 9709 raft_consensus.cc:1215] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250627 13:54:14.493557 11679 tablet_copy_client.cc:806] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:14.493948 11679 tablet_copy_client.cc:670] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:14.499593 7926 consensus_queue.cc:237] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:14.508602 11679 tablet_copy_client.cc:538] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:14.512295 11679 tablet_bootstrap.cc:492] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157: Bootstrap starting.
I20250627 13:54:14.515828 11635 tablet_bootstrap.cc:492] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:14.516484 11635 tablet_bootstrap.cc:492] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe: Bootstrap complete.
I20250627 13:54:14.516834 11635 ts_tablet_manager.cc:1397] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe: Time spent bootstrapping tablet: real 0.220s user 0.053s sys 0.030s
I20250627 13:54:14.517674 11635 raft_consensus.cc:357] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:14.519666 11635 raft_consensus.cc:738] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6686c6c365bf4f4582058890161676fe, State: Initialized, Role: LEARNER
I20250627 13:54:14.523241 9842 raft_consensus.cc:1273] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:14.523738 8061 raft_consensus.cc:1273] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20250627 13:54:14.524694 7869 consensus_peers.cc:487] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 -> Peer 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101): Couldn't send request to peer 2a1282ebe92a4c9a92eed3b13f69012a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cff879c8e57d42c28440b3bd310f1ba9. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:14.525995 7801 raft_consensus.cc:1273] T cff879c8e57d42c28440b3bd310f1ba9 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:14.530012 11336 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:14.530531 11336 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:14.533705 11021 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:14.546720 11636 tablet_bootstrap.cc:492] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:14.547191 11636 tablet_bootstrap.cc:492] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe: Bootstrap complete.
I20250627 13:54:14.547492 11636 ts_tablet_manager.cc:1397] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe: Time spent bootstrapping tablet: real 0.278s user 0.056s sys 0.005s
I20250627 13:54:14.548942 11636 raft_consensus.cc:357] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:14.549546 11636 raft_consensus.cc:738] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6686c6c365bf4f4582058890161676fe, State: Initialized, Role: LEARNER
I20250627 13:54:14.549818 11636 consensus_queue.cc:260] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:14.561926 11635 consensus_queue.cc:260] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:14.575588 11021 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:14.577366 11635 ts_tablet_manager.cc:1428] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe: Time spent starting tablet: real 0.059s user 0.003s sys 0.000s
I20250627 13:54:14.579562 8213 tablet_copy_service.cc:342] P e4c831df4da841929d13d6c783099b0e: Request end of tablet copy session 6686c6c365bf4f4582058890161676fe-8ab02f13e45c48d293e2d2d229d11e27 received from {username='slave'} at 127.0.103.71:35925
I20250627 13:54:14.579764 8213 tablet_copy_service.cc:434] P e4c831df4da841929d13d6c783099b0e: ending tablet copy session 6686c6c365bf4f4582058890161676fe-8ab02f13e45c48d293e2d2d229d11e27 on tablet 8ab02f13e45c48d293e2d2d229d11e27 with peer 6686c6c365bf4f4582058890161676fe
I20250627 13:54:14.584194 11336 raft_consensus.cc:1062] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854: attempting to promote NON_VOTER 956ff058e6d64579a1963aabdb6d466b to VOTER
I20250627 13:54:14.584908 11336 consensus_queue.cc:237] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:14.586778 7653 catalog_manager.cc:5582] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index 4 to 5, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250627 13:54:14.589139 7869 consensus_peers.cc:487] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 -> Peer 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101): Couldn't send request to peer 2a1282ebe92a4c9a92eed3b13f69012a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cff879c8e57d42c28440b3bd310f1ba9. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:14.589866 9840 raft_consensus.cc:1273] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:14.590446 11555 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:14.591580 9840 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:14.594854 7801 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:14.595645 8061 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:14.616453 8061 raft_consensus.cc:1273] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:14.616577 7801 raft_consensus.cc:1273] T cff879c8e57d42c28440b3bd310f1ba9 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:14.617141 11336 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:14.617962 11700 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250627 13:54:14.620183 11021 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:14.626119 7801 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:14.629195 11636 ts_tablet_manager.cc:1428] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe: Time spent starting tablet: real 0.082s user 0.033s sys 0.046s
I20250627 13:54:14.629921 8213 tablet_copy_service.cc:342] P e4c831df4da841929d13d6c783099b0e: Request end of tablet copy session 6686c6c365bf4f4582058890161676fe-e3053b529a264e8697a1df9e17eeea81 received from {username='slave'} at 127.0.103.71:35925
I20250627 13:54:14.630098 8213 tablet_copy_service.cc:434] P e4c831df4da841929d13d6c783099b0e: ending tablet copy session 6686c6c365bf4f4582058890161676fe-e3053b529a264e8697a1df9e17eeea81 on tablet e3053b529a264e8697a1df9e17eeea81 with peer 6686c6c365bf4f4582058890161676fe
I20250627 13:54:14.634259 9842 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:14.634519 8061 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:14.638197 11378 raft_consensus.cc:1062] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER 956ff058e6d64579a1963aabdb6d466b to VOTER
I20250627 13:54:14.638599 7654 catalog_manager.cc:5582] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index 5 to 6, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:14.639225 11378 consensus_queue.cc:237] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:14.653853 7926 consensus_queue.cc:237] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
W20250627 13:54:14.656730 7869 consensus_peers.cc:487] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 -> Peer 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101): Couldn't send request to peer 2a1282ebe92a4c9a92eed3b13f69012a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cff879c8e57d42c28440b3bd310f1ba9. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:14.657598 8062 raft_consensus.cc:1273] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:14.658308 11021 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:14.661016 11555 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:14.667769 7647 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet cff879c8e57d42c28440b3bd310f1ba9 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:14.669057 8062 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:14.671849 9842 raft_consensus.cc:1273] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:14.672446 11555 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:14.672523 7655 catalog_manager.cc:5582] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index 6 to 7, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:14.679033 9842 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:14.679373 7782 tablet_service.cc:1515] Processing DeleteTablet for tablet cff879c8e57d42c28440b3bd310f1ba9 with delete_type TABLET_DATA_TOMBSTONED (TS 83574c1bf8ea45adaff939158f93c096 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45826
I20250627 13:54:14.679723 11663 tablet_replica.cc:331] T cff879c8e57d42c28440b3bd310f1ba9 P 83574c1bf8ea45adaff939158f93c096: stopping tablet replica
I20250627 13:54:14.679913 11663 raft_consensus.cc:2241] T cff879c8e57d42c28440b3bd310f1ba9 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:14.680222 11663 raft_consensus.cc:2270] T cff879c8e57d42c28440b3bd310f1ba9 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:14.682075 11663 ts_tablet_manager.cc:1905] T cff879c8e57d42c28440b3bd310f1ba9 P 83574c1bf8ea45adaff939158f93c096: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:14.687930 11663 ts_tablet_manager.cc:1918] T cff879c8e57d42c28440b3bd310f1ba9 P 83574c1bf8ea45adaff939158f93c096: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20250627 13:54:14.688077 11663 log.cc:1199] T cff879c8e57d42c28440b3bd310f1ba9 P 83574c1bf8ea45adaff939158f93c096: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/wals/cff879c8e57d42c28440b3bd310f1ba9
I20250627 13:54:14.688874 7645 catalog_manager.cc:4928] TS 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467): tablet cff879c8e57d42c28440b3bd310f1ba9 (table rebalance_test_table_4 [id=4d92d465970b4ec4b5eace263b25362a]) successfully deleted
I20250627 13:54:14.691049 8189 raft_consensus.cc:1273] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:14.691800 11382 consensus_queue.cc:1035] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:14.693467 9840 raft_consensus.cc:1273] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:14.693941 7801 raft_consensus.cc:1273] T 9196497cb11f4ccbbd62695253d4c8c9 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:14.695787 11382 consensus_queue.cc:1035] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:14.696178 11188 consensus_queue.cc:1035] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:14.709308 8061 consensus_queue.cc:237] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:14.716393 9973 raft_consensus.cc:1215] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250627 13:54:14.748777 11636 ts_tablet_manager.cc:927] T 91664403eed74badbeba83d69f597e1f P 6686c6c365bf4f4582058890161676fe: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
W20250627 13:54:14.752566 7999 consensus_peers.cc:487] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 6686c6c365bf4f4582058890161676fe (127.0.103.71:37859): Couldn't send request to peer 6686c6c365bf4f4582058890161676fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2468f4746224f0ab6f4ba4b3a628e35. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:14.753491 8193 raft_consensus.cc:1273] T c2468f4746224f0ab6f4ba4b3a628e35 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:14.754045 11188 consensus_queue.cc:1035] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:14.754567 11378 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:14.755981 11636 tablet_copy_client.cc:323] T 91664403eed74badbeba83d69f597e1f P 6686c6c365bf4f4582058890161676fe: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:14.761346 8189 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:14.761777 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet 91664403eed74badbeba83d69f597e1f from peer 6686c6c365bf4f4582058890161676fe ({username='slave'} at 127.0.103.71:59981)
I20250627 13:54:14.761921 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet 91664403eed74badbeba83d69f597e1f from peer 6686c6c365bf4f4582058890161676fe at {username='slave'} at 127.0.103.71:59981: session id = 6686c6c365bf4f4582058890161676fe-91664403eed74badbeba83d69f597e1f
I20250627 13:54:14.762009 7800 raft_consensus.cc:1273] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:14.763041 11382 consensus_queue.cc:1035] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:14.763770 7800 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:14.767149 8082 tablet_copy_source_session.cc:215] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:14.770309 8190 raft_consensus.cc:2953] T c2468f4746224f0ab6f4ba4b3a628e35 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:14.771168 7654 catalog_manager.cc:5582] T 9196497cb11f4ccbbd62695253d4c8c9 P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 4 to 5, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:14.773635 11636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91664403eed74badbeba83d69f597e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:14.769261 11382 raft_consensus.cc:2953] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:14.786638 7654 catalog_manager.cc:5582] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index -1 to 4, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:14.786928 8061 consensus_queue.cc:237] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:14.795094 8062 consensus_queue.cc:237] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:14.798208 11679 tablet_bootstrap.cc:492] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:14.798607 11679 tablet_bootstrap.cc:492] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157: Bootstrap complete.
I20250627 13:54:14.798856 11679 ts_tablet_manager.cc:1397] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157: Time spent bootstrapping tablet: real 0.287s user 0.055s sys 0.002s
I20250627 13:54:14.799623 7800 raft_consensus.cc:2953] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:14.801056 11636 tablet_copy_client.cc:806] T 91664403eed74badbeba83d69f597e1f P 6686c6c365bf4f4582058890161676fe: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:14.801548 11636 tablet_copy_client.cc:670] T 91664403eed74badbeba83d69f597e1f P 6686c6c365bf4f4582058890161676fe: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:14.803874 11679 raft_consensus.cc:357] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:14.804083 11679 raft_consensus.cc:738] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dede81a8c0164427a61532d01b6fc157, State: Initialized, Role: LEARNER
I20250627 13:54:14.804239 11679 consensus_queue.cc:260] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:14.805163 11636 tablet_copy_client.cc:538] T 91664403eed74badbeba83d69f597e1f P 6686c6c365bf4f4582058890161676fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:14.805617 11624 ts_tablet_manager.cc:927] T 00feb28e9047435dbe36a595b1e23fdd P 956ff058e6d64579a1963aabdb6d466b: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:14.806190 11624 tablet_copy_client.cc:323] T 00feb28e9047435dbe36a595b1e23fdd P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:14.806896 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet 00feb28e9047435dbe36a595b1e23fdd from peer 956ff058e6d64579a1963aabdb6d466b ({username='slave'} at 127.0.103.70:59981)
I20250627 13:54:14.807045 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet 00feb28e9047435dbe36a595b1e23fdd from peer 956ff058e6d64579a1963aabdb6d466b at {username='slave'} at 127.0.103.70:59981: session id = 956ff058e6d64579a1963aabdb6d466b-00feb28e9047435dbe36a595b1e23fdd
I20250627 13:54:14.811230 11679 ts_tablet_manager.cc:1428] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157: Time spent starting tablet: real 0.012s user 0.002s sys 0.003s
I20250627 13:54:14.811897 8213 tablet_copy_service.cc:342] P e4c831df4da841929d13d6c783099b0e: Request end of tablet copy session dede81a8c0164427a61532d01b6fc157-5f176b66fdce4398be3492a5f3b4ea27 received from {username='slave'} at 127.0.103.72:33995
I20250627 13:54:14.812065 8213 tablet_copy_service.cc:434] P e4c831df4da841929d13d6c783099b0e: ending tablet copy session dede81a8c0164427a61532d01b6fc157-5f176b66fdce4398be3492a5f3b4ea27 on tablet 5f176b66fdce4398be3492a5f3b4ea27 with peer dede81a8c0164427a61532d01b6fc157
I20250627 13:54:14.813028 8082 tablet_copy_source_session.cc:215] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:14.814106 8193 raft_consensus.cc:1273] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:14.817538 11624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00feb28e9047435dbe36a595b1e23fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:14.817754 11378 consensus_queue.cc:1035] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:14.818909 9840 raft_consensus.cc:1273] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:14.819394 11378 consensus_queue.cc:1035] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:14.819527 9842 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:14.829190 11636 tablet_bootstrap.cc:492] T 91664403eed74badbeba83d69f597e1f P 6686c6c365bf4f4582058890161676fe: Bootstrap starting.
W20250627 13:54:14.834491 7996 consensus_peers.cc:487] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 118d5145b23f43798e1c727f2a9e9f92. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:14.835114 8193 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:14.835747 7926 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:14.836347 11382 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:14.837340 11188 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:14.837210 11382 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:14.837620 9842 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:14.841961 11188 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:14.845153 7926 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:14.846113 8193 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:14.853478 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9196497cb11f4ccbbd62695253d4c8c9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:14.854993 7654 catalog_manager.cc:5582] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index -1 to 4, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:14.859684 11624 tablet_copy_client.cc:806] T 00feb28e9047435dbe36a595b1e23fdd P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:14.860291 11624 tablet_copy_client.cc:670] T 00feb28e9047435dbe36a595b1e23fdd P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:14.862505 11624 tablet_copy_client.cc:538] T 00feb28e9047435dbe36a595b1e23fdd P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:14.868671 11664 raft_consensus.cc:1062] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096: attempting to promote NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a to VOTER
I20250627 13:54:14.869825 11664 consensus_queue.cc:237] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:14.871256 11624 tablet_bootstrap.cc:492] T 00feb28e9047435dbe36a595b1e23fdd P 956ff058e6d64579a1963aabdb6d466b: Bootstrap starting.
I20250627 13:54:14.887665 7654 catalog_manager.cc:5582] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 5 to 6, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:14.894397 8189 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:14.897748 7782 tablet_service.cc:1515] Processing DeleteTablet for tablet 9196497cb11f4ccbbd62695253d4c8c9 with delete_type TABLET_DATA_TOMBSTONED (TS 83574c1bf8ea45adaff939158f93c096 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45826
I20250627 13:54:14.898181 11663 tablet_replica.cc:331] T 9196497cb11f4ccbbd62695253d4c8c9 P 83574c1bf8ea45adaff939158f93c096: stopping tablet replica
I20250627 13:54:14.898428 11663 raft_consensus.cc:2241] T 9196497cb11f4ccbbd62695253d4c8c9 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:14.898722 11663 raft_consensus.cc:2270] T 9196497cb11f4ccbbd62695253d4c8c9 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:14.899766 11663 ts_tablet_manager.cc:1905] T 9196497cb11f4ccbbd62695253d4c8c9 P 83574c1bf8ea45adaff939158f93c096: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:14.909631 11663 ts_tablet_manager.cc:1918] T 9196497cb11f4ccbbd62695253d4c8c9 P 83574c1bf8ea45adaff939158f93c096: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250627 13:54:14.910543 9709 raft_consensus.cc:1273] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:14.913008 11663 log.cc:1199] T 9196497cb11f4ccbbd62695253d4c8c9 P 83574c1bf8ea45adaff939158f93c096: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/wals/9196497cb11f4ccbbd62695253d4c8c9
I20250627 13:54:14.911689 11273 consensus_queue.cc:1035] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:14.914978 7645 catalog_manager.cc:4928] TS 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467): tablet 9196497cb11f4ccbbd62695253d4c8c9 (table rebalance_test_table_4 [id=4d92d465970b4ec4b5eace263b25362a]) successfully deleted
I20250627 13:54:14.915325 7926 raft_consensus.cc:1273] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:14.916114 8062 raft_consensus.cc:1273] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:14.916940 11665 consensus_queue.cc:1035] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:14.917538 11665 consensus_queue.cc:1035] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:14.966631 10103 raft_consensus.cc:1215] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250627 13:54:14.984403 11665 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:14.993036 7655 catalog_manager.cc:5582] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:14.999379 9973 raft_consensus.cc:1215] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250627 13:54:15.000583 7933 consensus_queue.cc:237] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } }
I20250627 13:54:15.005900 8062 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:15.016819 9711 raft_consensus.cc:1273] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:15.019024 11555 consensus_queue.cc:1035] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:15.020519 8189 raft_consensus.cc:1273] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20250627 13:54:15.021201 7866 consensus_peers.cc:487] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 -> Peer 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467): Couldn't send request to peer 83574c1bf8ea45adaff939158f93c096. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:15.021533 11555 consensus_queue.cc:1035] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:15.040678 7933 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:15.048336 11555 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } }
I20250627 13:54:15.057636 9711 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } }
I20250627 13:54:15.067873 7655 catalog_manager.cc:5582] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index 6 to 7, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:15.076406 9711 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:15.083415 7801 consensus_queue.cc:237] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:15.086175 9709 raft_consensus.cc:1273] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:15.086648 11664 consensus_queue.cc:1035] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:15.088352 7926 raft_consensus.cc:1273] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:15.088959 11664 consensus_queue.cc:1035] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:15.085757 8193 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } }
I20250627 13:54:15.094851 11776 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:15.098932 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 1dc569a7b12d40058d073f7b81b2601c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:15.100828 9709 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:15.105370 7926 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:15.107882 11490 raft_consensus.cc:1062] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: attempting to promote NON_VOTER 6686c6c365bf4f4582058890161676fe to VOTER
I20250627 13:54:15.108083 7655 catalog_manager.cc:5582] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 5 to 6, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:15.108964 11490 consensus_queue.cc:237] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:15.115723 8041 tablet_service.cc:1515] Processing DeleteTablet for tablet 1dc569a7b12d40058d073f7b81b2601c with delete_type TABLET_DATA_TOMBSTONED (TS 9cf479bdcd0242738d2a77db4ad9f660 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59760
I20250627 13:54:15.120326 11798 tablet_replica.cc:331] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660: stopping tablet replica
I20250627 13:54:15.122149 11798 raft_consensus.cc:2241] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:15.122733 9973 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:15.123310 11493 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:15.126848 8062 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:15.127131 11798 raft_consensus.cc:2270] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:15.127429 11493 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:15.128652 11798 ts_tablet_manager.cc:1905] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:15.129665 9709 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:15.136826 11492 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:15.137360 11492 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:15.140178 8062 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:15.144459 9973 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:15.146494 9709 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:15.150436 7652 catalog_manager.cc:5582] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 7 to 8, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:15.153256 11798 ts_tablet_manager.cc:1918] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250627 13:54:15.153539 11798 log.cc:1199] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal/wals/1dc569a7b12d40058d073f7b81b2601c
I20250627 13:54:15.154834 7645 catalog_manager.cc:4928] TS 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933): tablet 1dc569a7b12d40058d073f7b81b2601c (table rebalance_test_table_3 [id=ec6a9ed6ab4d416988b2abe2f8d29c55]) successfully deleted
I20250627 13:54:15.171406 7801 consensus_queue.cc:237] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:15.175083 11812 ts_tablet_manager.cc:927] T c2468f4746224f0ab6f4ba4b3a628e35 P 6686c6c365bf4f4582058890161676fe: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:15.182344 11812 tablet_copy_client.cc:323] T c2468f4746224f0ab6f4ba4b3a628e35 P 6686c6c365bf4f4582058890161676fe: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:15.183561 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet c2468f4746224f0ab6f4ba4b3a628e35 from peer 6686c6c365bf4f4582058890161676fe ({username='slave'} at 127.0.103.71:59981)
I20250627 13:54:15.183708 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet c2468f4746224f0ab6f4ba4b3a628e35 from peer 6686c6c365bf4f4582058890161676fe at {username='slave'} at 127.0.103.71:59981: session id = 6686c6c365bf4f4582058890161676fe-c2468f4746224f0ab6f4ba4b3a628e35
I20250627 13:54:15.187623 8082 tablet_copy_source_session.cc:215] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:15.189576 11812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2468f4746224f0ab6f4ba4b3a628e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:15.206344 9709 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:15.206352 7926 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:15.206954 11772 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:15.207618 11757 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:15.208964 11812 tablet_copy_client.cc:806] T c2468f4746224f0ab6f4ba4b3a628e35 P 6686c6c365bf4f4582058890161676fe: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:15.209374 11812 tablet_copy_client.cc:670] T c2468f4746224f0ab6f4ba4b3a628e35 P 6686c6c365bf4f4582058890161676fe: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:15.215176 11812 tablet_copy_client.cc:538] T c2468f4746224f0ab6f4ba4b3a628e35 P 6686c6c365bf4f4582058890161676fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:15.218834 11812 tablet_bootstrap.cc:492] T c2468f4746224f0ab6f4ba4b3a628e35 P 6686c6c365bf4f4582058890161676fe: Bootstrap starting.
W20250627 13:54:15.223438 7738 consensus_peers.cc:487] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 -> Peer 6686c6c365bf4f4582058890161676fe (127.0.103.71:37859): Couldn't send request to peer 6686c6c365bf4f4582058890161676fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87726824b05a4b968386f57d3a7a4aad. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:15.230566 11624 tablet_bootstrap.cc:492] T 00feb28e9047435dbe36a595b1e23fdd P 956ff058e6d64579a1963aabdb6d466b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:15.236676 11624 tablet_bootstrap.cc:492] T 00feb28e9047435dbe36a595b1e23fdd P 956ff058e6d64579a1963aabdb6d466b: Bootstrap complete.
I20250627 13:54:15.236981 11624 ts_tablet_manager.cc:1397] T 00feb28e9047435dbe36a595b1e23fdd P 956ff058e6d64579a1963aabdb6d466b: Time spent bootstrapping tablet: real 0.366s user 0.064s sys 0.019s
I20250627 13:54:15.239090 11624 raft_consensus.cc:357] T 00feb28e9047435dbe36a595b1e23fdd P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:15.239287 11624 raft_consensus.cc:738] T 00feb28e9047435dbe36a595b1e23fdd P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 956ff058e6d64579a1963aabdb6d466b, State: Initialized, Role: LEARNER
I20250627 13:54:15.239506 11624 consensus_queue.cc:260] T 00feb28e9047435dbe36a595b1e23fdd P 956ff058e6d64579a1963aabdb6d466b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:15.242122 11624 ts_tablet_manager.cc:1428] T 00feb28e9047435dbe36a595b1e23fdd P 956ff058e6d64579a1963aabdb6d466b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250627 13:54:15.242790 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session 956ff058e6d64579a1963aabdb6d466b-00feb28e9047435dbe36a595b1e23fdd received from {username='slave'} at 127.0.103.70:59981
I20250627 13:54:15.242985 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session 956ff058e6d64579a1963aabdb6d466b-00feb28e9047435dbe36a595b1e23fdd on tablet 00feb28e9047435dbe36a595b1e23fdd with peer 956ff058e6d64579a1963aabdb6d466b
I20250627 13:54:15.251868 11636 tablet_bootstrap.cc:492] T 91664403eed74badbeba83d69f597e1f P 6686c6c365bf4f4582058890161676fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:15.251845 11664 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:15.252324 11636 tablet_bootstrap.cc:492] T 91664403eed74badbeba83d69f597e1f P 6686c6c365bf4f4582058890161676fe: Bootstrap complete.
I20250627 13:54:15.252573 11636 ts_tablet_manager.cc:1397] T 91664403eed74badbeba83d69f597e1f P 6686c6c365bf4f4582058890161676fe: Time spent bootstrapping tablet: real 0.435s user 0.051s sys 0.006s
I20250627 13:54:15.252440 9710 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:15.253301 11636 raft_consensus.cc:357] T 91664403eed74badbeba83d69f597e1f P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:15.253482 11636 raft_consensus.cc:738] T 91664403eed74badbeba83d69f597e1f P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6686c6c365bf4f4582058890161676fe, State: Initialized, Role: LEARNER
I20250627 13:54:15.253736 11636 consensus_queue.cc:260] T 91664403eed74badbeba83d69f597e1f P 6686c6c365bf4f4582058890161676fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:15.254397 11636 ts_tablet_manager.cc:1428] T 91664403eed74badbeba83d69f597e1f P 6686c6c365bf4f4582058890161676fe: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20250627 13:54:15.255096 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session 6686c6c365bf4f4582058890161676fe-91664403eed74badbeba83d69f597e1f received from {username='slave'} at 127.0.103.71:59981
I20250627 13:54:15.255234 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session 6686c6c365bf4f4582058890161676fe-91664403eed74badbeba83d69f597e1f on tablet 91664403eed74badbeba83d69f597e1f with peer 6686c6c365bf4f4582058890161676fe
I20250627 13:54:15.260735 7655 catalog_manager.cc:5582] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 6 to 7, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:15.281633 11817 ts_tablet_manager.cc:927] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a: Initiating tablet copy from peer a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:54:15.287474 7926 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:15.307329 8062 consensus_queue.cc:237] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:15.311401 8193 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:15.312000 11382 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250627 13:54:15.313277 7996 consensus_peers.cc:487] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 118d5145b23f43798e1c727f2a9e9f92. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:15.316076 11746 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:15.321552 8193 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
W20250627 13:54:15.327630 7999 consensus_peers.cc:487] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 6686c6c365bf4f4582058890161676fe (127.0.103.71:37859): Couldn't send request to peer 6686c6c365bf4f4582058890161676fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 118d5145b23f43798e1c727f2a9e9f92. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:15.332602 7926 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:15.333175 11746 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:15.332927 7656 catalog_manager.cc:5582] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 4 to 5, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:15.335584 7926 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:15.362058 11817 tablet_copy_client.cc:323] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:43675
I20250627 13:54:15.362958 7953 tablet_copy_service.cc:140] P a41a89ae8ed743edb2b5cf4490d8c854: Received BeginTabletCopySession request for tablet cff879c8e57d42c28440b3bd310f1ba9 from peer 2a1282ebe92a4c9a92eed3b13f69012a ({username='slave'} at 127.0.103.69:49083)
I20250627 13:54:15.363143 7953 tablet_copy_service.cc:161] P a41a89ae8ed743edb2b5cf4490d8c854: Beginning new tablet copy session on tablet cff879c8e57d42c28440b3bd310f1ba9 from peer 2a1282ebe92a4c9a92eed3b13f69012a at {username='slave'} at 127.0.103.69:49083: session id = 2a1282ebe92a4c9a92eed3b13f69012a-cff879c8e57d42c28440b3bd310f1ba9
I20250627 13:54:15.366199 7953 tablet_copy_source_session.cc:215] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:15.368068 11817 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cff879c8e57d42c28440b3bd310f1ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:15.395942 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:15.396987 11817 tablet_copy_client.cc:806] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:15.397270 11817 tablet_copy_client.cc:670] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:15.405022 11492 raft_consensus.cc:1062] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e: attempting to promote NON_VOTER 6686c6c365bf4f4582058890161676fe to VOTER
I20250627 13:54:15.406201 11492 consensus_queue.cc:237] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:15.412612 8193 consensus_queue.cc:237] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:15.420262 10103 raft_consensus.cc:1273] T 6f5e370b77cc42fdb0b886ea19af06c3 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20250627 13:54:15.422251 8129 consensus_peers.cc:487] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e -> Peer 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101): Couldn't send request to peer 2a1282ebe92a4c9a92eed3b13f69012a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f5e370b77cc42fdb0b886ea19af06c3. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:15.423666 11817 tablet_copy_client.cc:538] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:15.425709 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:15.426311 8062 raft_consensus.cc:1273] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:15.427556 9973 raft_consensus.cc:1273] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:15.428130 11493 consensus_queue.cc:1035] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:15.428730 11493 consensus_queue.cc:1035] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:15.432001 11490 consensus_queue.cc:1035] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:15.432775 11817 tablet_bootstrap.cc:492] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap starting.
I20250627 13:54:15.449906 11844 ts_tablet_manager.cc:927] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096: Initiating tablet copy from peer a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:54:15.471994 11844 tablet_copy_client.cc:323] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:43675
I20250627 13:54:15.472187 11493 raft_consensus.cc:1062] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e: attempting to promote NON_VOTER dede81a8c0164427a61532d01b6fc157 to VOTER
I20250627 13:54:15.472893 7953 tablet_copy_service.cc:140] P a41a89ae8ed743edb2b5cf4490d8c854: Received BeginTabletCopySession request for tablet b265cfda4bfe48908659208ec8eeab2d from peer 83574c1bf8ea45adaff939158f93c096 ({username='slave'} at 127.0.103.65:51265)
I20250627 13:54:15.473057 7953 tablet_copy_service.cc:161] P a41a89ae8ed743edb2b5cf4490d8c854: Beginning new tablet copy session on tablet b265cfda4bfe48908659208ec8eeab2d from peer 83574c1bf8ea45adaff939158f93c096 at {username='slave'} at 127.0.103.65:51265: session id = 83574c1bf8ea45adaff939158f93c096-b265cfda4bfe48908659208ec8eeab2d
I20250627 13:54:15.476642 7953 tablet_copy_source_session.cc:215] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:15.478468 11493 consensus_queue.cc:237] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } }
I20250627 13:54:15.481293 11844 ts_tablet_manager.cc:1905] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250627 13:54:15.515312 11817 tablet_bootstrap.cc:492] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:15.515861 11817 tablet_bootstrap.cc:492] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap complete.
I20250627 13:54:15.516319 11817 ts_tablet_manager.cc:1397] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a: Time spent bootstrapping tablet: real 0.084s user 0.047s sys 0.010s
I20250627 13:54:15.518566 11817 raft_consensus.cc:357] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:15.518815 11817 raft_consensus.cc:738] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a1282ebe92a4c9a92eed3b13f69012a, State: Initialized, Role: LEARNER
I20250627 13:54:15.518994 11817 consensus_queue.cc:260] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:15.519616 11817 ts_tablet_manager.cc:1428] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250627 13:54:15.521117 7953 tablet_copy_service.cc:342] P a41a89ae8ed743edb2b5cf4490d8c854: Request end of tablet copy session 2a1282ebe92a4c9a92eed3b13f69012a-cff879c8e57d42c28440b3bd310f1ba9 received from {username='slave'} at 127.0.103.69:49083
I20250627 13:54:15.521325 7953 tablet_copy_service.cc:434] P a41a89ae8ed743edb2b5cf4490d8c854: ending tablet copy session 2a1282ebe92a4c9a92eed3b13f69012a-cff879c8e57d42c28440b3bd310f1ba9 on tablet cff879c8e57d42c28440b3bd310f1ba9 with peer 2a1282ebe92a4c9a92eed3b13f69012a
I20250627 13:54:15.547528 11812 tablet_bootstrap.cc:492] T c2468f4746224f0ab6f4ba4b3a628e35 P 6686c6c365bf4f4582058890161676fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:15.548132 11812 tablet_bootstrap.cc:492] T c2468f4746224f0ab6f4ba4b3a628e35 P 6686c6c365bf4f4582058890161676fe: Bootstrap complete.
I20250627 13:54:15.548635 11812 ts_tablet_manager.cc:1397] T c2468f4746224f0ab6f4ba4b3a628e35 P 6686c6c365bf4f4582058890161676fe: Time spent bootstrapping tablet: real 0.330s user 0.052s sys 0.002s
I20250627 13:54:15.549576 11812 raft_consensus.cc:357] T c2468f4746224f0ab6f4ba4b3a628e35 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:15.549856 11812 raft_consensus.cc:738] T c2468f4746224f0ab6f4ba4b3a628e35 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6686c6c365bf4f4582058890161676fe, State: Initialized, Role: LEARNER
I20250627 13:54:15.550161 11812 consensus_queue.cc:260] T c2468f4746224f0ab6f4ba4b3a628e35 P 6686c6c365bf4f4582058890161676fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:15.550840 11812 ts_tablet_manager.cc:1428] T c2468f4746224f0ab6f4ba4b3a628e35 P 6686c6c365bf4f4582058890161676fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:54:15.552376 7926 raft_consensus.cc:1273] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:15.553692 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session 6686c6c365bf4f4582058890161676fe-c2468f4746224f0ab6f4ba4b3a628e35 received from {username='slave'} at 127.0.103.71:59981
I20250627 13:54:15.553860 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session 6686c6c365bf4f4582058890161676fe-c2468f4746224f0ab6f4ba4b3a628e35 on tablet c2468f4746224f0ab6f4ba4b3a628e35 with peer 6686c6c365bf4f4582058890161676fe
I20250627 13:54:15.559393 11860 raft_consensus.cc:2953] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:15.559953 11847 consensus_queue.cc:1035] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:15.566072 8061 raft_consensus.cc:1273] T 5f176b66fdce4398be3492a5f3b4ea27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:15.567649 7656 catalog_manager.cc:5582] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 6 to 7, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:15.567929 11852 consensus_queue.cc:1035] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:15.571396 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:15.580374 8062 raft_consensus.cc:1273] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:15.581368 11492 consensus_queue.cc:1035] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:15.586287 10103 raft_consensus.cc:1273] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:15.587190 10103 raft_consensus.cc:2953] T 6f5e370b77cc42fdb0b886ea19af06c3 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:15.592955 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:15.593508 11847 consensus_queue.cc:1035] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250627 13:54:15.605383 7801 raft_consensus.cc:1273] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:15.605952 11492 consensus_queue.cc:1035] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:15.617549 11812 ts_tablet_manager.cc:927] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe: Initiating tablet copy from peer 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:54:15.631713 9711 raft_consensus.cc:1215] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250627 13:54:15.639190 11812 tablet_copy_client.cc:323] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:37467
I20250627 13:54:15.640027 7822 tablet_copy_service.cc:140] P 83574c1bf8ea45adaff939158f93c096: Received BeginTabletCopySession request for tablet 87726824b05a4b968386f57d3a7a4aad from peer 6686c6c365bf4f4582058890161676fe ({username='slave'} at 127.0.103.71:53529)
I20250627 13:54:15.640244 7822 tablet_copy_service.cc:161] P 83574c1bf8ea45adaff939158f93c096: Beginning new tablet copy session on tablet 87726824b05a4b968386f57d3a7a4aad from peer 6686c6c365bf4f4582058890161676fe at {username='slave'} at 127.0.103.71:53529: session id = 6686c6c365bf4f4582058890161676fe-87726824b05a4b968386f57d3a7a4aad
I20250627 13:54:15.643749 7822 tablet_copy_source_session.cc:215] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:15.645481 11492 raft_consensus.cc:2953] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:15.652145 7801 raft_consensus.cc:2953] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:15.659615 11812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87726824b05a4b968386f57d3a7a4aad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:15.664326 7655 catalog_manager.cc:5582] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 4 to 5, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:15.668241 10104 raft_consensus.cc:2953] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:15.674201 11860 raft_consensus.cc:2953] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:15.678297 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:15.678982 9972 raft_consensus.cc:2953] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:15.688983 7654 catalog_manager.cc:5582] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe reported cstate change: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:15.700771 8062 raft_consensus.cc:2953] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:15.702962 9973 raft_consensus.cc:1215] T c2468f4746224f0ab6f4ba4b3a628e35 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250627 13:54:15.709197 8063 raft_consensus.cc:2953] T 5f176b66fdce4398be3492a5f3b4ea27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:15.716451 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:15.720635 8193 consensus_queue.cc:237] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:15.725694 9972 raft_consensus.cc:1273] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:15.726608 11852 consensus_queue.cc:1035] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:15.729377 8060 raft_consensus.cc:2953] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:15.731737 11847 raft_consensus.cc:2953] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:15.733191 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:15.734185 9972 raft_consensus.cc:2953] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:15.734753 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:15.739945 7655 catalog_manager.cc:5582] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe reported cstate change: config changed from index 5 to 6, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:15.741812 11636 ts_tablet_manager.cc:927] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:15.757503 7648 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 8ab02f13e45c48d293e2d2d229d11e27 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:15.771334 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:15.771894 9840 raft_consensus.cc:1215] T 00feb28e9047435dbe36a595b1e23fdd P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250627 13:54:15.780398 9972 raft_consensus.cc:1215] T 91664403eed74badbeba83d69f597e1f P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20250627 13:54:15.784395 11844 ts_tablet_manager.cc:1918] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20250627 13:54:15.784693 11844 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b265cfda4bfe48908659208ec8eeab2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:15.787338 7911 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ab02f13e45c48d293e2d2d229d11e27 with delete_type TABLET_DATA_TOMBSTONED (TS a41a89ae8ed743edb2b5cf4490d8c854 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59302
I20250627 13:54:15.788411 11636 tablet_copy_client.cc:323] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:15.789072 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet 118d5145b23f43798e1c727f2a9e9f92 from peer 6686c6c365bf4f4582058890161676fe ({username='slave'} at 127.0.103.71:59981)
I20250627 13:54:15.789249 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet 118d5145b23f43798e1c727f2a9e9f92 from peer 6686c6c365bf4f4582058890161676fe at {username='slave'} at 127.0.103.71:59981: session id = 6686c6c365bf4f4582058890161676fe-118d5145b23f43798e1c727f2a9e9f92
I20250627 13:54:15.792776 8082 tablet_copy_source_session.cc:215] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:15.794245 11636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 118d5145b23f43798e1c727f2a9e9f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:15.794335 8063 raft_consensus.cc:1273] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:15.806527 11636 tablet_copy_client.cc:806] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:15.806527 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:15.806814 11636 tablet_copy_client.cc:670] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:15.808523 11490 consensus_queue.cc:1035] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:15.808856 11636 tablet_copy_client.cc:538] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:15.810215 10084 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:15.811403 8063 raft_consensus.cc:2953] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:15.819914 11636 tablet_bootstrap.cc:492] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe: Bootstrap starting.
I20250627 13:54:15.841094 11903 tablet_replica.cc:331] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854: stopping tablet replica
I20250627 13:54:15.841298 11903 raft_consensus.cc:2241] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:15.841840 11903 pending_rounds.cc:70] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854: Trying to abort 1 pending ops.
I20250627 13:54:15.841907 11903 pending_rounds.cc:77] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7172228937340305408 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "8ab02f13e45c48d293e2d2d229d11e27" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } } }
I20250627 13:54:15.842236 11903 raft_consensus.cc:2915] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20250627 13:54:15.842355 11903 raft_consensus.cc:2887] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250627 13:54:15.842447 11903 raft_consensus.cc:2270] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:15.843592 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:15.846185 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:15.855633 11903 ts_tablet_manager.cc:1905] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:15.857787 11812 tablet_copy_client.cc:806] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:15.858078 11812 tablet_copy_client.cc:670] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:15.860527 11812 tablet_copy_client.cc:538] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:15.863004 11903 ts_tablet_manager.cc:1918] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250627 13:54:15.863190 11903 log.cc:1199] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/wals/8ab02f13e45c48d293e2d2d229d11e27
I20250627 13:54:15.863883 11844 tablet_copy_client.cc:806] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:15.864205 11844 tablet_copy_client.cc:670] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:15.866252 11844 tablet_copy_client.cc:538] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:15.874226 11905 ts_tablet_manager.cc:927] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:15.877593 10084 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:15.878304 7647 catalog_manager.cc:4928] TS a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675): tablet 8ab02f13e45c48d293e2d2d229d11e27 (table rebalance_test_table_5 [id=dd79cdfc35164ae6a649d3a77ee397c2]) successfully deleted
I20250627 13:54:15.894769 11812 tablet_bootstrap.cc:492] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe: Bootstrap starting.
I20250627 13:54:15.899243 11844 tablet_bootstrap.cc:492] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:54:15.903577 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:15.906246 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:15.917801 7801 consensus_queue.cc:237] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
W20250627 13:54:15.933172 7737 consensus_peers.cc:487] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 -> Peer dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213): Couldn't send request to peer dede81a8c0164427a61532d01b6fc157. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c322cfebf12c4fef8d88fbd68e7e8b6f. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:15.934655 9711 raft_consensus.cc:1273] T c322cfebf12c4fef8d88fbd68e7e8b6f P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:15.935200 11757 consensus_queue.cc:1035] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:15.943894 7926 raft_consensus.cc:1273] T c322cfebf12c4fef8d88fbd68e7e8b6f P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:15.944476 11757 consensus_queue.cc:1035] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:15.982131 11664 raft_consensus.cc:2953] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:15.982832 7926 raft_consensus.cc:2953] T c322cfebf12c4fef8d88fbd68e7e8b6f P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:15.995011 7655 catalog_manager.cc:5582] T c322cfebf12c4fef8d88fbd68e7e8b6f P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index 6 to 7, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:16.021744 9709 raft_consensus.cc:2953] T c322cfebf12c4fef8d88fbd68e7e8b6f P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:16.046491 11921 ts_tablet_manager.cc:927] T 6f5e370b77cc42fdb0b886ea19af06c3 P 2a1282ebe92a4c9a92eed3b13f69012a: Initiating tablet copy from peer e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:54:16.048111 11905 tablet_copy_client.cc:323] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:16.049185 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet 118d5145b23f43798e1c727f2a9e9f92 from peer 956ff058e6d64579a1963aabdb6d466b ({username='slave'} at 127.0.103.70:59981)
I20250627 13:54:16.049398 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet 118d5145b23f43798e1c727f2a9e9f92 from peer 956ff058e6d64579a1963aabdb6d466b at {username='slave'} at 127.0.103.70:59981: session id = 956ff058e6d64579a1963aabdb6d466b-118d5145b23f43798e1c727f2a9e9f92
I20250627 13:54:16.053442 8082 tablet_copy_source_session.cc:215] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:16.055284 11905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 118d5145b23f43798e1c727f2a9e9f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:16.076602 11905 tablet_copy_client.cc:806] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:16.076990 11905 tablet_copy_client.cc:670] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:16.078660 11905 tablet_copy_client.cc:538] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:16.085624 11921 tablet_copy_client.cc:323] T 6f5e370b77cc42fdb0b886ea19af06c3 P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.68:46139
I20250627 13:54:16.086333 8213 tablet_copy_service.cc:140] P e4c831df4da841929d13d6c783099b0e: Received BeginTabletCopySession request for tablet 6f5e370b77cc42fdb0b886ea19af06c3 from peer 2a1282ebe92a4c9a92eed3b13f69012a ({username='slave'} at 127.0.103.69:43551)
I20250627 13:54:16.086527 8213 tablet_copy_service.cc:161] P e4c831df4da841929d13d6c783099b0e: Beginning new tablet copy session on tablet 6f5e370b77cc42fdb0b886ea19af06c3 from peer 2a1282ebe92a4c9a92eed3b13f69012a at {username='slave'} at 127.0.103.69:43551: session id = 2a1282ebe92a4c9a92eed3b13f69012a-6f5e370b77cc42fdb0b886ea19af06c3
I20250627 13:54:16.090180 8213 tablet_copy_source_session.cc:215] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:16.091574 11921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f5e370b77cc42fdb0b886ea19af06c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:16.100922 11921 tablet_copy_client.cc:806] T 6f5e370b77cc42fdb0b886ea19af06c3 P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:16.101336 11921 tablet_copy_client.cc:670] T 6f5e370b77cc42fdb0b886ea19af06c3 P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:16.103358 11921 tablet_copy_client.cc:538] T 6f5e370b77cc42fdb0b886ea19af06c3 P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:16.117158 11921 tablet_bootstrap.cc:492] T 6f5e370b77cc42fdb0b886ea19af06c3 P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap starting.
W20250627 13:54:16.122232 7737 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.66:43675, user_credentials={real_user=slave}} blocked reactor thread for 97702.2us
I20250627 13:54:16.126552 11905 tablet_bootstrap.cc:492] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap starting.
I20250627 13:54:16.150404 8062 consensus_queue.cc:237] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
W20250627 13:54:16.163709 7996 consensus_peers.cc:487] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 644985e3086d4614b6598a727cb60c1d. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:16.164911 10103 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:16.165117 8193 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:16.165588 11382 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:16.166236 11382 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:16.186973 11382 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:16.190471 10104 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:16.192142 8189 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:16.197187 7655 catalog_manager.cc:5582] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157 reported cstate change: config changed from index 6 to 7, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:16.229259 11921 tablet_bootstrap.cc:492] T 6f5e370b77cc42fdb0b886ea19af06c3 P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:16.229744 11921 tablet_bootstrap.cc:492] T 6f5e370b77cc42fdb0b886ea19af06c3 P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap complete.
I20250627 13:54:16.230023 11921 ts_tablet_manager.cc:1397] T 6f5e370b77cc42fdb0b886ea19af06c3 P 2a1282ebe92a4c9a92eed3b13f69012a: Time spent bootstrapping tablet: real 0.113s user 0.041s sys 0.016s
I20250627 13:54:16.230777 11921 raft_consensus.cc:357] T 6f5e370b77cc42fdb0b886ea19af06c3 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:16.230991 11921 raft_consensus.cc:738] T 6f5e370b77cc42fdb0b886ea19af06c3 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a1282ebe92a4c9a92eed3b13f69012a, State: Initialized, Role: LEARNER
I20250627 13:54:16.231333 11921 consensus_queue.cc:260] T 6f5e370b77cc42fdb0b886ea19af06c3 P 2a1282ebe92a4c9a92eed3b13f69012a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:16.232045 11921 ts_tablet_manager.cc:1428] T 6f5e370b77cc42fdb0b886ea19af06c3 P 2a1282ebe92a4c9a92eed3b13f69012a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250627 13:54:16.233165 8213 tablet_copy_service.cc:342] P e4c831df4da841929d13d6c783099b0e: Request end of tablet copy session 2a1282ebe92a4c9a92eed3b13f69012a-6f5e370b77cc42fdb0b886ea19af06c3 received from {username='slave'} at 127.0.103.69:43551
I20250627 13:54:16.233297 8213 tablet_copy_service.cc:434] P e4c831df4da841929d13d6c783099b0e: ending tablet copy session 2a1282ebe92a4c9a92eed3b13f69012a-6f5e370b77cc42fdb0b886ea19af06c3 on tablet 6f5e370b77cc42fdb0b886ea19af06c3 with peer 2a1282ebe92a4c9a92eed3b13f69012a
I20250627 13:54:16.236538 11382 raft_consensus.cc:1062] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER 956ff058e6d64579a1963aabdb6d466b to VOTER
I20250627 13:54:16.237448 11382 consensus_queue.cc:237] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:16.253528 9840 raft_consensus.cc:1273] T 00feb28e9047435dbe36a595b1e23fdd P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:16.254022 7801 raft_consensus.cc:1273] T 00feb28e9047435dbe36a595b1e23fdd P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:16.254810 11382 consensus_queue.cc:1035] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:16.256386 8193 raft_consensus.cc:1273] T 00feb28e9047435dbe36a595b1e23fdd P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:16.257194 11382 consensus_queue.cc:1035] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:16.259151 11746 consensus_queue.cc:1035] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250627 13:54:16.275185 11905 log.cc:927] Time spent T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: Append to log took a long time: real 0.093s user 0.000s sys 0.000s
I20250627 13:54:16.281472 11746 raft_consensus.cc:2953] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:16.285799 11940 raft_consensus.cc:1062] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854: attempting to promote NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a to VOTER
I20250627 13:54:16.286329 8193 raft_consensus.cc:2953] T 00feb28e9047435dbe36a595b1e23fdd P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:16.286713 11940 consensus_queue.cc:237] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:16.292537 7653 catalog_manager.cc:5582] T 00feb28e9047435dbe36a595b1e23fdd P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 4 to 5, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:16.300097 9711 raft_consensus.cc:1273] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:16.301160 11555 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:16.306252 11746 raft_consensus.cc:1062] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER 6686c6c365bf4f4582058890161676fe to VOTER
I20250627 13:54:16.306769 9840 raft_consensus.cc:1273] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:16.307126 11746 consensus_queue.cc:237] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:16.308055 8062 raft_consensus.cc:1273] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:16.309243 9840 raft_consensus.cc:2953] T 00feb28e9047435dbe36a595b1e23fdd P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:16.309599 11938 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:16.309676 7801 raft_consensus.cc:2953] T 00feb28e9047435dbe36a595b1e23fdd P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:16.310629 11938 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:16.320549 8062 consensus_queue.cc:237] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:16.323832 8193 raft_consensus.cc:1273] T 00feb28e9047435dbe36a595b1e23fdd P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:16.324435 11954 consensus_queue.cc:1035] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:16.325397 11954 raft_consensus.cc:1062] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER 6686c6c365bf4f4582058890161676fe to VOTER
I20250627 13:54:16.326769 11954 consensus_queue.cc:237] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:16.330945 11378 raft_consensus.cc:2953] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:16.336655 8193 raft_consensus.cc:2953] T 00feb28e9047435dbe36a595b1e23fdd P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:16.346665 7654 catalog_manager.cc:5582] T 00feb28e9047435dbe36a595b1e23fdd P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 5 to 6, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:16.351035 7782 tablet_service.cc:1515] Processing DeleteTablet for tablet 00feb28e9047435dbe36a595b1e23fdd with delete_type TABLET_DATA_TOMBSTONED (TS 83574c1bf8ea45adaff939158f93c096 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45826
I20250627 13:54:16.351615 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 00feb28e9047435dbe36a595b1e23fdd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:16.352010 9840 raft_consensus.cc:1273] T 00feb28e9047435dbe36a595b1e23fdd P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:16.353340 11378 consensus_queue.cc:1035] T 00feb28e9047435dbe36a595b1e23fdd P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:16.354741 9840 raft_consensus.cc:2953] T 00feb28e9047435dbe36a595b1e23fdd P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:16.359524 8193 raft_consensus.cc:1273] T c2468f4746224f0ab6f4ba4b3a628e35 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:16.360085 11746 consensus_queue.cc:1035] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:16.362181 7801 raft_consensus.cc:1273] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:16.362692 11378 consensus_queue.cc:1035] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:16.368695 9972 raft_consensus.cc:1273] T c2468f4746224f0ab6f4ba4b3a628e35 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:16.372810 11956 consensus_queue.cc:1035] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:16.375641 11382 raft_consensus.cc:2953] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:16.383360 8189 raft_consensus.cc:2953] T c2468f4746224f0ab6f4ba4b3a628e35 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:16.385356 9972 raft_consensus.cc:1273] T 91664403eed74badbeba83d69f597e1f P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:16.385911 11746 consensus_queue.cc:1035] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:16.390053 7655 catalog_manager.cc:5582] T c2468f4746224f0ab6f4ba4b3a628e35 P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:16.390259 11959 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:16.406164 9711 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:16.410378 9973 raft_consensus.cc:2953] T c2468f4746224f0ab6f4ba4b3a628e35 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:16.416026 11968 tablet_replica.cc:331] T 00feb28e9047435dbe36a595b1e23fdd P 83574c1bf8ea45adaff939158f93c096: stopping tablet replica
I20250627 13:54:16.416325 11968 raft_consensus.cc:2241] T 00feb28e9047435dbe36a595b1e23fdd P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:16.416653 11968 raft_consensus.cc:2270] T 00feb28e9047435dbe36a595b1e23fdd P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:16.399917 7801 raft_consensus.cc:2953] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:16.421957 8062 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:16.428042 10103 raft_consensus.cc:1273] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:16.428997 7926 raft_consensus.cc:1273] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:16.430986 11746 consensus_queue.cc:1035] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:16.431712 11746 consensus_queue.cc:1035] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:16.441814 11961 raft_consensus.cc:2953] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:16.442802 11968 ts_tablet_manager.cc:1905] T 00feb28e9047435dbe36a595b1e23fdd P 83574c1bf8ea45adaff939158f93c096: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:16.446367 7926 raft_consensus.cc:2953] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:16.453219 9840 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:16.457526 10103 raft_consensus.cc:2953] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:16.457872 9972 raft_consensus.cc:2953] T 91664403eed74badbeba83d69f597e1f P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:16.463374 7655 catalog_manager.cc:5582] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index 7 to 8, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:16.465153 7656 catalog_manager.cc:5582] T 91664403eed74badbeba83d69f597e1f P 6686c6c365bf4f4582058890161676fe reported cstate change: config changed from index 8 to 9, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:16.504725 11844 tablet_bootstrap.cc:492] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:16.505312 11844 tablet_bootstrap.cc:492] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096: Bootstrap complete.
I20250627 13:54:16.519114 9711 raft_consensus.cc:1215] T 6f5e370b77cc42fdb0b886ea19af06c3 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250627 13:54:16.522611 11844 ts_tablet_manager.cc:1397] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.606s user 0.049s sys 0.005s
I20250627 13:54:16.524291 11844 raft_consensus.cc:357] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } }
I20250627 13:54:16.524554 11844 raft_consensus.cc:738] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: LEARNER
I20250627 13:54:16.524775 11844 consensus_queue.cc:260] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } }
I20250627 13:54:16.529019 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:16.559891 11968 ts_tablet_manager.cc:1918] T 00feb28e9047435dbe36a595b1e23fdd P 83574c1bf8ea45adaff939158f93c096: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250627 13:54:16.560324 11968 log.cc:1199] T 00feb28e9047435dbe36a595b1e23fdd P 83574c1bf8ea45adaff939158f93c096: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/wals/00feb28e9047435dbe36a595b1e23fdd
I20250627 13:54:16.561606 7645 catalog_manager.cc:4928] TS 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467): tablet 00feb28e9047435dbe36a595b1e23fdd (table rebalance_test_table_8 [id=b84f3fe3fd7c4f01902547d2db982f90]) successfully deleted
I20250627 13:54:16.597975 11844 ts_tablet_manager.cc:1428] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.075s user 0.002s sys 0.000s
I20250627 13:54:16.606729 7953 tablet_copy_service.cc:342] P a41a89ae8ed743edb2b5cf4490d8c854: Request end of tablet copy session 83574c1bf8ea45adaff939158f93c096-b265cfda4bfe48908659208ec8eeab2d received from {username='slave'} at 127.0.103.65:51265
I20250627 13:54:16.606907 7953 tablet_copy_service.cc:434] P a41a89ae8ed743edb2b5cf4490d8c854: ending tablet copy session 83574c1bf8ea45adaff939158f93c096-b265cfda4bfe48908659208ec8eeab2d on tablet b265cfda4bfe48908659208ec8eeab2d with peer 83574c1bf8ea45adaff939158f93c096
I20250627 13:54:16.609700 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:16.628023 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:16.652825 12002 ts_tablet_manager.cc:927] T c322cfebf12c4fef8d88fbd68e7e8b6f P dede81a8c0164427a61532d01b6fc157: Initiating tablet copy from peer 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:54:16.663049 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:16.676451 12016 ts_tablet_manager.cc:927] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:16.699618 12002 tablet_copy_client.cc:323] T c322cfebf12c4fef8d88fbd68e7e8b6f P dede81a8c0164427a61532d01b6fc157: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:37467
I20250627 13:54:16.700825 7822 tablet_copy_service.cc:140] P 83574c1bf8ea45adaff939158f93c096: Received BeginTabletCopySession request for tablet c322cfebf12c4fef8d88fbd68e7e8b6f from peer dede81a8c0164427a61532d01b6fc157 ({username='slave'} at 127.0.103.72:38213)
I20250627 13:54:16.701009 7822 tablet_copy_service.cc:161] P 83574c1bf8ea45adaff939158f93c096: Beginning new tablet copy session on tablet c322cfebf12c4fef8d88fbd68e7e8b6f from peer dede81a8c0164427a61532d01b6fc157 at {username='slave'} at 127.0.103.72:38213: session id = dede81a8c0164427a61532d01b6fc157-c322cfebf12c4fef8d88fbd68e7e8b6f
I20250627 13:54:16.704447 7822 tablet_copy_source_session.cc:215] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:16.707283 10084 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:16.711622 12002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c322cfebf12c4fef8d88fbd68e7e8b6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:16.726913 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:16.735852 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:16.738219 12016 tablet_copy_client.cc:323] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:16.739066 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet 644985e3086d4614b6598a727cb60c1d from peer 956ff058e6d64579a1963aabdb6d466b ({username='slave'} at 127.0.103.70:59981)
I20250627 13:54:16.739244 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet 644985e3086d4614b6598a727cb60c1d from peer 956ff058e6d64579a1963aabdb6d466b at {username='slave'} at 127.0.103.70:59981: session id = 956ff058e6d64579a1963aabdb6d466b-644985e3086d4614b6598a727cb60c1d
I20250627 13:54:16.742390 8082 tablet_copy_source_session.cc:215] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:16.744000 12016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 644985e3086d4614b6598a727cb60c1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:16.774572 12002 tablet_copy_client.cc:806] T c322cfebf12c4fef8d88fbd68e7e8b6f P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:16.774960 12002 tablet_copy_client.cc:670] T c322cfebf12c4fef8d88fbd68e7e8b6f P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:16.777786 11636 tablet_bootstrap.cc:492] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:16.778364 11636 tablet_bootstrap.cc:492] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe: Bootstrap complete.
I20250627 13:54:16.778645 11812 tablet_bootstrap.cc:492] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:16.779071 11636 ts_tablet_manager.cc:1397] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe: Time spent bootstrapping tablet: real 0.959s user 0.062s sys 0.009s
I20250627 13:54:16.780730 12002 tablet_copy_client.cc:538] T c322cfebf12c4fef8d88fbd68e7e8b6f P dede81a8c0164427a61532d01b6fc157: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:16.784947 12002 tablet_bootstrap.cc:492] T c322cfebf12c4fef8d88fbd68e7e8b6f P dede81a8c0164427a61532d01b6fc157: Bootstrap starting.
I20250627 13:54:16.790161 11812 tablet_bootstrap.cc:492] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe: Bootstrap complete.
I20250627 13:54:16.790506 11812 ts_tablet_manager.cc:1397] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe: Time spent bootstrapping tablet: real 0.896s user 0.059s sys 0.000s
I20250627 13:54:16.791277 11636 raft_consensus.cc:357] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:16.793054 11636 raft_consensus.cc:738] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6686c6c365bf4f4582058890161676fe, State: Initialized, Role: LEARNER
I20250627 13:54:16.799978 12016 tablet_copy_client.cc:806] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:16.800285 12016 tablet_copy_client.cc:670] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:16.805945 12016 tablet_copy_client.cc:538] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:16.814843 11812 raft_consensus.cc:357] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:16.815265 11812 raft_consensus.cc:738] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6686c6c365bf4f4582058890161676fe, State: Initialized, Role: LEARNER
I20250627 13:54:16.815577 11812 consensus_queue.cc:260] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:16.816491 11812 ts_tablet_manager.cc:1428] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20250627 13:54:16.817764 7822 tablet_copy_service.cc:342] P 83574c1bf8ea45adaff939158f93c096: Request end of tablet copy session 6686c6c365bf4f4582058890161676fe-87726824b05a4b968386f57d3a7a4aad received from {username='slave'} at 127.0.103.71:53529
I20250627 13:54:16.817916 7822 tablet_copy_service.cc:434] P 83574c1bf8ea45adaff939158f93c096: ending tablet copy session 6686c6c365bf4f4582058890161676fe-87726824b05a4b968386f57d3a7a4aad on tablet 87726824b05a4b968386f57d3a7a4aad with peer 6686c6c365bf4f4582058890161676fe
I20250627 13:54:16.826233 12016 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: Bootstrap starting.
I20250627 13:54:16.828615 11636 consensus_queue.cc:260] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:16.832329 11636 ts_tablet_manager.cc:1428] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe: Time spent starting tablet: real 0.053s user 0.002s sys 0.000s
I20250627 13:54:16.856082 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session 6686c6c365bf4f4582058890161676fe-118d5145b23f43798e1c727f2a9e9f92 received from {username='slave'} at 127.0.103.71:59981
I20250627 13:54:16.856292 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session 6686c6c365bf4f4582058890161676fe-118d5145b23f43798e1c727f2a9e9f92 on tablet 118d5145b23f43798e1c727f2a9e9f92 with peer 6686c6c365bf4f4582058890161676fe
W20250627 13:54:16.858932 8129 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.69:33101, user_credentials={real_user=slave}} blocked reactor thread for 49845us
I20250627 13:54:16.873565 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:16.917524 11905 tablet_bootstrap.cc:492] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:16.918028 11905 tablet_bootstrap.cc:492] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap complete.
I20250627 13:54:16.918321 11905 ts_tablet_manager.cc:1397] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: Time spent bootstrapping tablet: real 0.799s user 0.051s sys 0.011s
I20250627 13:54:16.919186 11905 raft_consensus.cc:357] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:16.919405 11905 raft_consensus.cc:738] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 956ff058e6d64579a1963aabdb6d466b, State: Initialized, Role: LEARNER
I20250627 13:54:16.919745 11905 consensus_queue.cc:260] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:16.945726 11905 ts_tablet_manager.cc:1428] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20250627 13:54:16.946400 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session 956ff058e6d64579a1963aabdb6d466b-118d5145b23f43798e1c727f2a9e9f92 received from {username='slave'} at 127.0.103.70:59981
I20250627 13:54:16.946543 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session 956ff058e6d64579a1963aabdb6d466b-118d5145b23f43798e1c727f2a9e9f92 on tablet 118d5145b23f43798e1c727f2a9e9f92 with peer 956ff058e6d64579a1963aabdb6d466b
I20250627 13:54:16.961584 7801 raft_consensus.cc:1215] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250627 13:54:16.975652 12040 raft_consensus.cc:1062] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e: attempting to promote NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a to VOTER
I20250627 13:54:16.976866 12040 consensus_queue.cc:237] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:16.992403 9711 raft_consensus.cc:1273] T 6f5e370b77cc42fdb0b886ea19af06c3 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:16.993085 11490 consensus_queue.cc:1035] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:17.003604 10103 raft_consensus.cc:1273] T 6f5e370b77cc42fdb0b886ea19af06c3 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:17.005493 11490 consensus_queue.cc:1035] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:17.025322 12002 tablet_bootstrap.cc:492] T c322cfebf12c4fef8d88fbd68e7e8b6f P dede81a8c0164427a61532d01b6fc157: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:17.025805 12002 tablet_bootstrap.cc:492] T c322cfebf12c4fef8d88fbd68e7e8b6f P dede81a8c0164427a61532d01b6fc157: Bootstrap complete.
I20250627 13:54:17.026088 12002 ts_tablet_manager.cc:1397] T c322cfebf12c4fef8d88fbd68e7e8b6f P dede81a8c0164427a61532d01b6fc157: Time spent bootstrapping tablet: real 0.241s user 0.052s sys 0.015s
I20250627 13:54:17.026861 12002 raft_consensus.cc:357] T c322cfebf12c4fef8d88fbd68e7e8b6f P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:17.027091 12002 raft_consensus.cc:738] T c322cfebf12c4fef8d88fbd68e7e8b6f P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dede81a8c0164427a61532d01b6fc157, State: Initialized, Role: LEARNER
I20250627 13:54:17.027307 12002 consensus_queue.cc:260] T c322cfebf12c4fef8d88fbd68e7e8b6f P dede81a8c0164427a61532d01b6fc157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:17.034812 12002 ts_tablet_manager.cc:1428] T c322cfebf12c4fef8d88fbd68e7e8b6f P dede81a8c0164427a61532d01b6fc157: Time spent starting tablet: real 0.009s user 0.000s sys 0.007s
I20250627 13:54:17.036599 7822 tablet_copy_service.cc:342] P 83574c1bf8ea45adaff939158f93c096: Request end of tablet copy session dede81a8c0164427a61532d01b6fc157-c322cfebf12c4fef8d88fbd68e7e8b6f received from {username='slave'} at 127.0.103.72:38213
I20250627 13:54:17.036767 7822 tablet_copy_service.cc:434] P 83574c1bf8ea45adaff939158f93c096: ending tablet copy session dede81a8c0164427a61532d01b6fc157-c322cfebf12c4fef8d88fbd68e7e8b6f on tablet c322cfebf12c4fef8d88fbd68e7e8b6f with peer dede81a8c0164427a61532d01b6fc157
I20250627 13:54:17.042796 11490 raft_consensus.cc:2953] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:17.051476 10104 raft_consensus.cc:2953] T 6f5e370b77cc42fdb0b886ea19af06c3 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:17.051999 9711 raft_consensus.cc:2953] T 6f5e370b77cc42fdb0b886ea19af06c3 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:17.055984 9973 raft_consensus.cc:1215] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250627 13:54:17.062922 7655 catalog_manager.cc:5582] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 7 to 8, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:17.080412 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:17.106848 8062 raft_consensus.cc:1273] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:17.107048 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:17.107662 12040 consensus_queue.cc:1035] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:17.134315 8062 raft_consensus.cc:2953] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:17.147887 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:17.178056 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:17.226114 10104 raft_consensus.cc:1215] T c322cfebf12c4fef8d88fbd68e7e8b6f P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250627 13:54:17.229943 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:17.259126 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:17.261250 10084 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:17.306640 9973 raft_consensus.cc:1215] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20250627 13:54:17.311300 9840 raft_consensus.cc:1215] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20250627 13:54:17.314052 12016 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:17.314584 12016 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: Bootstrap complete.
I20250627 13:54:17.314878 12016 ts_tablet_manager.cc:1397] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: Time spent bootstrapping tablet: real 0.489s user 0.051s sys 0.005s
I20250627 13:54:17.315667 12016 raft_consensus.cc:357] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:17.315889 12016 raft_consensus.cc:738] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 956ff058e6d64579a1963aabdb6d466b, State: Initialized, Role: LEARNER
I20250627 13:54:17.316126 12016 consensus_queue.cc:260] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:17.318252 12016 ts_tablet_manager.cc:1428] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20250627 13:54:17.319142 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session 956ff058e6d64579a1963aabdb6d466b-644985e3086d4614b6598a727cb60c1d received from {username='slave'} at 127.0.103.70:59981
I20250627 13:54:17.319322 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session 956ff058e6d64579a1963aabdb6d466b-644985e3086d4614b6598a727cb60c1d on tablet 644985e3086d4614b6598a727cb60c1d with peer 956ff058e6d64579a1963aabdb6d466b
I20250627 13:54:17.331517 8189 consensus_queue.cc:237] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:17.334379 9711 raft_consensus.cc:1273] T 6f5e370b77cc42fdb0b886ea19af06c3 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:17.335172 12040 consensus_queue.cc:1035] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:17.338471 10104 raft_consensus.cc:1273] T 6f5e370b77cc42fdb0b886ea19af06c3 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:17.339313 12040 consensus_queue.cc:1035] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:17.341048 12040 raft_consensus.cc:2953] T 6f5e370b77cc42fdb0b886ea19af06c3 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:17.344566 9711 raft_consensus.cc:2953] T 6f5e370b77cc42fdb0b886ea19af06c3 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:17.355062 7657 catalog_manager.cc:5582] T 6f5e370b77cc42fdb0b886ea19af06c3 P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: config changed from index 8 to 9, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:17.356995 10104 raft_consensus.cc:2953] T 6f5e370b77cc42fdb0b886ea19af06c3 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:17.365890 7648 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 6f5e370b77cc42fdb0b886ea19af06c3 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:17.396994 8041 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f5e370b77cc42fdb0b886ea19af06c3 with delete_type TABLET_DATA_TOMBSTONED (TS 9cf479bdcd0242738d2a77db4ad9f660 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:59760
I20250627 13:54:17.398653 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:17.427546 12104 tablet_replica.cc:331] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660: stopping tablet replica
I20250627 13:54:17.427909 12104 raft_consensus.cc:2241] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:17.428519 12104 raft_consensus.cc:2270] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:17.438205 12104 ts_tablet_manager.cc:1905] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:17.443090 11959 raft_consensus.cc:1062] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854: attempting to promote NON_VOTER 83574c1bf8ea45adaff939158f93c096 to VOTER
I20250627 13:54:17.444015 11959 consensus_queue.cc:237] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } }
I20250627 13:54:17.455965 12104 ts_tablet_manager.cc:1918] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20250627 13:54:17.460427 12104 log.cc:1199] T 6f5e370b77cc42fdb0b886ea19af06c3 P 9cf479bdcd0242738d2a77db4ad9f660: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal/wals/6f5e370b77cc42fdb0b886ea19af06c3
I20250627 13:54:17.461598 7645 catalog_manager.cc:4928] TS 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933): tablet 6f5e370b77cc42fdb0b886ea19af06c3 (table rebalance_test_table_3 [id=ec6a9ed6ab4d416988b2abe2f8d29c55]) successfully deleted
I20250627 13:54:17.473834 7801 raft_consensus.cc:1273] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 [term 1 LEARNER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:17.474576 11940 consensus_queue.cc:1035] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:17.477921 9711 raft_consensus.cc:1273] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:17.478713 11940 consensus_queue.cc:1035] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:17.491699 11920 raft_consensus.cc:1062] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096: attempting to promote NON_VOTER 6686c6c365bf4f4582058890161676fe to VOTER
I20250627 13:54:17.492499 11920 consensus_queue.cc:237] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:17.523231 7933 consensus_queue.cc:237] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
W20250627 13:54:17.530624 7869 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.69:33101, user_credentials={real_user=slave}} blocked reactor thread for 52136.9us
I20250627 13:54:17.531678 8189 raft_consensus.cc:1273] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:17.540784 9973 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:17.543457 12088 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:17.553431 9709 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:17.554327 12088 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:17.557869 7926 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:17.558904 12082 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:17.562667 12088 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:17.571955 9709 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:17.582490 7653 catalog_manager.cc:5582] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: config changed from index 7 to 8, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:17.604467 7926 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:17.630319 11940 consensus_queue.cc:1035] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:17.659363 11940 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:17.660190 9973 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:17.668877 7801 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:17.670814 7800 consensus_queue.cc:237] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:17.676611 9709 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:17.676741 8189 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:17.677253 12083 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:17.680645 7660 catalog_manager.cc:5582] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 7 to 8, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:17.682668 9973 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:17.683230 11920 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:17.686312 12088 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:17.687906 9972 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:17.693002 7656 catalog_manager.cc:5582] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe reported cstate change: config changed from index 8 to 9, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:17.695408 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 87726824b05a4b968386f57d3a7a4aad with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:17.697293 7911 tablet_service.cc:1515] Processing DeleteTablet for tablet 87726824b05a4b968386f57d3a7a4aad with delete_type TABLET_DATA_TOMBSTONED (TS a41a89ae8ed743edb2b5cf4490d8c854 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:59302
I20250627 13:54:17.697672 9709 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:17.713091 11382 raft_consensus.cc:1062] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER 6686c6c365bf4f4582058890161676fe to VOTER
I20250627 13:54:17.714218 11382 consensus_queue.cc:237] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:17.719123 12146 tablet_replica.cc:331] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854: stopping tablet replica
I20250627 13:54:17.719383 12146 raft_consensus.cc:2241] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:17.719697 12146 raft_consensus.cc:2270] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:17.726667 9840 raft_consensus.cc:1215] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250627 13:54:17.739692 9711 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:17.748268 12146 ts_tablet_manager.cc:1905] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:17.756007 12088 raft_consensus.cc:1062] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096: attempting to promote NON_VOTER dede81a8c0164427a61532d01b6fc157 to VOTER
I20250627 13:54:17.757031 12088 consensus_queue.cc:237] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } }
I20250627 13:54:17.762683 7801 raft_consensus.cc:1273] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:17.763449 11940 consensus_queue.cc:1035] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:17.783823 9972 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:17.784619 11961 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:17.787714 7801 raft_consensus.cc:2953] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:17.799341 9840 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:17.800257 11746 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:17.804751 9711 raft_consensus.cc:1273] T 72db4bf98ab04b7189640f658d8fdcd7 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:17.827803 8189 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:17.829602 10104 raft_consensus.cc:1273] T c322cfebf12c4fef8d88fbd68e7e8b6f P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:17.830611 12082 consensus_queue.cc:1035] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:17.838889 11382 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:17.839418 9711 raft_consensus.cc:1273] T c322cfebf12c4fef8d88fbd68e7e8b6f P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:17.842097 12088 consensus_queue.cc:1035] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:17.876189 11920 raft_consensus.cc:2953] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:17.882349 9711 raft_consensus.cc:2953] T c322cfebf12c4fef8d88fbd68e7e8b6f P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:17.889628 7657 catalog_manager.cc:5582] T c322cfebf12c4fef8d88fbd68e7e8b6f P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: config changed from index 7 to 8, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:17.890643 10104 raft_consensus.cc:2953] T c322cfebf12c4fef8d88fbd68e7e8b6f P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:17.897141 7660 catalog_manager.cc:5582] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 6 to 7, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:17.897825 11382 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:17.905124 8189 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:17.906266 7801 consensus_queue.cc:237] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } }
I20250627 13:54:17.909660 9709 raft_consensus.cc:1273] T c322cfebf12c4fef8d88fbd68e7e8b6f P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:17.910434 12082 consensus_queue.cc:1035] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:17.913305 7660 catalog_manager.cc:5582] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 5 to 6, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:17.918380 12082 raft_consensus.cc:2953] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:17.923645 8062 consensus_queue.cc:237] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:17.924391 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet c322cfebf12c4fef8d88fbd68e7e8b6f with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:17.926939 7653 catalog_manager.cc:5582] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 8 to 9, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:17.927392 9709 raft_consensus.cc:2953] T c322cfebf12c4fef8d88fbd68e7e8b6f P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:17.938715 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:17.938043 12115 consensus_queue.cc:1035] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.133s
I20250627 13:54:17.942549 11940 raft_consensus.cc:2953] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:17.948876 7926 raft_consensus.cc:1273] T c322cfebf12c4fef8d88fbd68e7e8b6f P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:17.953531 7931 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:17.953852 7927 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:17.953936 7911 tablet_service.cc:1515] Processing DeleteTablet for tablet c322cfebf12c4fef8d88fbd68e7e8b6f with delete_type TABLET_DATA_TOMBSTONED (TS a41a89ae8ed743edb2b5cf4490d8c854 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:59302
I20250627 13:54:17.954792 7926 consensus_queue.cc:237] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } }
I20250627 13:54:17.954880 11382 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:17.956606 7926 raft_consensus.cc:870] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
W20250627 13:54:17.956864 7926 raft_consensus.cc:1577] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:17.958712 9710 raft_consensus.cc:1273] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:17.959357 12115 consensus_queue.cc:1035] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:17.966478 7926 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:17.975986 12115 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:17.982581 7647 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet b265cfda4bfe48908659208ec8eeab2d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:17.987552 12146 ts_tablet_manager.cc:1918] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20250627 13:54:17.987778 12146 log.cc:1199] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/wals/87726824b05a4b968386f57d3a7a4aad
I20250627 13:54:17.988711 12146 tablet_replica.cc:331] T c322cfebf12c4fef8d88fbd68e7e8b6f P a41a89ae8ed743edb2b5cf4490d8c854: stopping tablet replica
I20250627 13:54:17.988756 7647 catalog_manager.cc:4928] TS a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675): tablet 87726824b05a4b968386f57d3a7a4aad (table rebalance_test_table_1 [id=8961aa0df2f1488f82cd55cfa22ad214]) successfully deleted
I20250627 13:54:17.988786 8189 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:17.988787 7660 catalog_manager.cc:5582] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index 8 to 9, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:17.991811 9711 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:17.993731 8171 tablet_service.cc:1515] Processing DeleteTablet for tablet b265cfda4bfe48908659208ec8eeab2d with delete_type TABLET_DATA_TOMBSTONED (TS e4c831df4da841929d13d6c783099b0e not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:48616
I20250627 13:54:17.999823 9972 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:18.011891 12194 tablet_replica.cc:331] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e: stopping tablet replica
I20250627 13:54:18.013048 12194 raft_consensus.cc:2241] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:18.015429 11382 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:18.020188 12146 raft_consensus.cc:2241] T c322cfebf12c4fef8d88fbd68e7e8b6f P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:18.021100 11961 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:18.022255 12146 raft_consensus.cc:2270] T c322cfebf12c4fef8d88fbd68e7e8b6f P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:18.023715 12146 ts_tablet_manager.cc:1905] T c322cfebf12c4fef8d88fbd68e7e8b6f P a41a89ae8ed743edb2b5cf4490d8c854: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:18.023830 12194 raft_consensus.cc:2270] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:18.029441 12194 ts_tablet_manager.cc:1905] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:18.030642 7927 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:18.038484 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 118d5145b23f43798e1c727f2a9e9f92 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:18.046285 8189 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:18.056792 7660 catalog_manager.cc:5582] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 6 to 7, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:18.065624 9822 tablet_service.cc:1515] Processing DeleteTablet for tablet 118d5145b23f43798e1c727f2a9e9f92 with delete_type TABLET_DATA_TOMBSTONED (TS 956ff058e6d64579a1963aabdb6d466b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55228
I20250627 13:54:18.099973 12194 ts_tablet_manager.cc:1918] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20250627 13:54:18.101444 12215 tablet_replica.cc:331] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: stopping tablet replica
I20250627 13:54:18.101696 12194 log.cc:1199] T b265cfda4bfe48908659208ec8eeab2d P e4c831df4da841929d13d6c783099b0e: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal/wals/b265cfda4bfe48908659208ec8eeab2d
I20250627 13:54:18.101779 12215 raft_consensus.cc:2241] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Raft consensus shutting down.
I20250627 13:54:18.102025 12215 pending_rounds.cc:70] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: Trying to abort 1 pending ops.
I20250627 13:54:18.102231 12215 pending_rounds.cc:77] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7172228946794594304 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "118d5145b23f43798e1c727f2a9e9f92" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } } }
I20250627 13:54:18.102831 7648 catalog_manager.cc:4928] TS e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): tablet b265cfda4bfe48908659208ec8eeab2d (table rebalance_test_table_0 [id=d8bfc9268a4a4c2c92bc20d56ac73086]) successfully deleted
I20250627 13:54:18.102988 12215 raft_consensus.cc:2915] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20250627 13:54:18.103196 12215 raft_consensus.cc:2887] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250627 13:54:18.103417 12215 raft_consensus.cc:2270] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Raft consensus is shut down!
W20250627 13:54:18.103530 7868 consensus_peers.cc:487] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 -> Peer dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213): Couldn't send request to peer dede81a8c0164427a61532d01b6fc157. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72db4bf98ab04b7189640f658d8fdcd7. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:18.120945 9709 raft_consensus.cc:2953] T 72db4bf98ab04b7189640f658d8fdcd7 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:18.123320 12146 ts_tablet_manager.cc:1918] T c322cfebf12c4fef8d88fbd68e7e8b6f P a41a89ae8ed743edb2b5cf4490d8c854: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20250627 13:54:18.123620 12146 log.cc:1199] T c322cfebf12c4fef8d88fbd68e7e8b6f P a41a89ae8ed743edb2b5cf4490d8c854: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/wals/c322cfebf12c4fef8d88fbd68e7e8b6f
I20250627 13:54:18.125819 7647 catalog_manager.cc:4928] TS a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675): tablet c322cfebf12c4fef8d88fbd68e7e8b6f (table rebalance_test_table_5 [id=dd79cdfc35164ae6a649d3a77ee397c2]) successfully deleted
I20250627 13:54:18.129053 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:18.122810 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:18.127965 9821 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:18.132442 12215 ts_tablet_manager.cc:1905] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:18.142761 8062 consensus_queue.cc:237] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:18.146335 9842 raft_consensus.cc:1273] T 570b134177314b31851951c411a9f4bf P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:18.146934 12180 consensus_queue.cc:1035] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:18.148541 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
W20250627 13:54:18.149739 7999 consensus_peers.cc:487] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101): Couldn't send request to peer 2a1282ebe92a4c9a92eed3b13f69012a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 570b134177314b31851951c411a9f4bf. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:18.152113 9973 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:18.154069 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:18.166693 11382 raft_consensus.cc:2953] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:18.167814 8189 raft_consensus.cc:1273] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:18.171998 9840 raft_consensus.cc:2953] T 570b134177314b31851951c411a9f4bf P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:18.174880 7656 catalog_manager.cc:5582] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 6 to 7, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:18.185279 11382 consensus_queue.cc:1035] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:18.197536 11382 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:18.199790 12180 raft_consensus.cc:1062] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER 956ff058e6d64579a1963aabdb6d466b to VOTER
I20250627 13:54:18.200696 12180 consensus_queue.cc:237] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
W20250627 13:54:18.205909 7737 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.66:43675, user_credentials={real_user=slave}} blocked reactor thread for 56664.4us
I20250627 13:54:18.207512 7801 raft_consensus.cc:1273] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:18.209698 9840 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:18.210237 12179 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:18.211602 11940 consensus_queue.cc:1035] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:18.214586 9973 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:18.214948 7801 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:18.229552 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:18.234723 10102 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:18.235656 12180 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:18.243793 8193 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:18.245939 12180 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:18.262362 8189 raft_consensus.cc:2953] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:18.274371 12179 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:18.276149 8190 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:18.276466 12215 ts_tablet_manager.cc:1918] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20250627 13:54:18.276599 12215 log.cc:1199] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal/wals/118d5145b23f43798e1c727f2a9e9f92
I20250627 13:54:18.278098 10103 raft_consensus.cc:1273] T c322cfebf12c4fef8d88fbd68e7e8b6f P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:18.278743 7645 catalog_manager.cc:4928] TS 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): tablet 118d5145b23f43798e1c727f2a9e9f92 (table rebalance_test_table_5 [id=dd79cdfc35164ae6a649d3a77ee397c2]) successfully deleted
I20250627 13:54:18.281236 9840 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:18.282102 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:18.282299 7653 catalog_manager.cc:5582] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 7 to 8, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
W20250627 13:54:18.285538 7736 consensus_peers.cc:159] SignalRequest failed: Illegal state: peer closed
I20250627 13:54:18.299876 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:18.307740 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:18.310379 12082 consensus_queue.cc:1035] T c322cfebf12c4fef8d88fbd68e7e8b6f P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:18.332162 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:18.346570 10103 raft_consensus.cc:2953] T c322cfebf12c4fef8d88fbd68e7e8b6f P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:18.349917 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:18.368105 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:18.372159 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:18.391858 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:18.394663 12238 ts_tablet_manager.cc:927] T 72db4bf98ab04b7189640f658d8fdcd7 P dede81a8c0164427a61532d01b6fc157: Initiating tablet copy from peer a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:54:18.415773 10084 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:18.415920 10083 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:18.419814 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:18.420974 12238 tablet_copy_client.cc:323] T 72db4bf98ab04b7189640f658d8fdcd7 P dede81a8c0164427a61532d01b6fc157: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:43675
I20250627 13:54:18.423027 8062 consensus_queue.cc:237] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:18.425523 9840 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:18.426120 12179 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:18.428360 11961 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:18.430608 9840 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:18.434265 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 644985e3086d4614b6598a727cb60c1d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:18.435952 7660 catalog_manager.cc:5582] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 8 to 9, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:18.417760 10102 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:18.441915 8171 tablet_service.cc:1515] Processing DeleteTablet for tablet 644985e3086d4614b6598a727cb60c1d with delete_type TABLET_DATA_TOMBSTONED (TS e4c831df4da841929d13d6c783099b0e not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:48616
I20250627 13:54:18.442250 12194 tablet_replica.cc:331] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: stopping tablet replica
I20250627 13:54:18.442478 12194 raft_consensus.cc:2241] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:18.442732 12194 raft_consensus.cc:2270] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:18.444288 12194 ts_tablet_manager.cc:1905] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:18.451920 12194 ts_tablet_manager.cc:1918] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20250627 13:54:18.452097 12194 log.cc:1199] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal/wals/644985e3086d4614b6598a727cb60c1d
I20250627 13:54:18.452883 7648 catalog_manager.cc:4928] TS e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): tablet 644985e3086d4614b6598a727cb60c1d (table rebalance_test_table_1 [id=8961aa0df2f1488f82cd55cfa22ad214]) successfully deleted
I20250627 13:54:18.459556 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:18.469166 8189 consensus_queue.cc:237] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:18.477563 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:18.478478 10083 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:18.489706 10104 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:18.493913 11961 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:18.500527 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:18.502871 9973 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:18.503569 11862 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:18.510293 9709 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:18.510949 11862 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:18.514834 10102 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:18.521060 8062 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:18.521725 11862 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:18.527829 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:18.528432 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:18.558393 12040 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:18.563812 7953 tablet_copy_service.cc:140] P a41a89ae8ed743edb2b5cf4490d8c854: Received BeginTabletCopySession request for tablet 72db4bf98ab04b7189640f658d8fdcd7 from peer dede81a8c0164427a61532d01b6fc157 ({username='slave'} at 127.0.103.72:34895)
I20250627 13:54:18.563997 7953 tablet_copy_service.cc:161] P a41a89ae8ed743edb2b5cf4490d8c854: Beginning new tablet copy session on tablet 72db4bf98ab04b7189640f658d8fdcd7 from peer dede81a8c0164427a61532d01b6fc157 at {username='slave'} at 127.0.103.72:34895: session id = dede81a8c0164427a61532d01b6fc157-72db4bf98ab04b7189640f658d8fdcd7
I20250627 13:54:18.566826 7953 tablet_copy_source_session.cc:215] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:18.569659 12238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72db4bf98ab04b7189640f658d8fdcd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:18.576730 8062 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:18.581764 9709 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:18.582464 7660 catalog_manager.cc:5582] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 8 to 9. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:18.585559 9973 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:18.595023 8189 consensus_queue.cc:237] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:18.615202 9711 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:18.616145 12258 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:18.620635 9973 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:18.626319 11862 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:18.631839 9711 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:18.635682 7648 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet e3053b529a264e8697a1df9e17eeea81 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:18.637100 7656 catalog_manager.cc:5582] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 9 to 10, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:18.639003 12238 tablet_copy_client.cc:806] T 72db4bf98ab04b7189640f658d8fdcd7 P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:18.639300 12238 tablet_copy_client.cc:670] T 72db4bf98ab04b7189640f658d8fdcd7 P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:18.640937 12258 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:18.641608 8041 tablet_service.cc:1515] Processing DeleteTablet for tablet e3053b529a264e8697a1df9e17eeea81 with delete_type TABLET_DATA_TOMBSTONED (TS 9cf479bdcd0242738d2a77db4ad9f660 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:59760
I20250627 13:54:18.642513 8189 consensus_queue.cc:237] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:18.646471 8063 consensus_queue.cc:237] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
W20250627 13:54:18.646799 8126 consensus_peers.cc:487] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e3053b529a264e8697a1df9e17eeea81. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:18.648959 9974 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:18.649683 12180 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:18.653234 8190 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:18.655416 9709 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:18.655357 12180 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:18.655942 11862 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:18.658984 9972 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:18.660933 12180 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:18.664422 12261 tablet_replica.cc:331] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660: stopping tablet replica
I20250627 13:54:18.665182 9974 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:18.666505 12261 raft_consensus.cc:2241] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:18.666836 12261 raft_consensus.cc:2270] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:18.667071 8190 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:18.667861 12261 ts_tablet_manager.cc:1905] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:18.672585 11862 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:18.675130 9972 raft_consensus.cc:870] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }:
New config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
W20250627 13:54:18.675264 9972 raft_consensus.cc:1577] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 11' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }:
New config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:18.675719 9972 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:18.681384 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 118d5145b23f43798e1c727f2a9e9f92 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:18.691663 12264 ts_tablet_manager.cc:927] T 570b134177314b31851951c411a9f4bf P 2a1282ebe92a4c9a92eed3b13f69012a: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:18.692116 7660 catalog_manager.cc:5582] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe reported cstate change: config changed from index 7 to 8, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:18.692449 11862 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:18.696568 7911 tablet_service.cc:1515] Processing DeleteTablet for tablet 118d5145b23f43798e1c727f2a9e9f92 with delete_type TABLET_DATA_TOMBSTONED (TS a41a89ae8ed743edb2b5cf4490d8c854 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:59302
I20250627 13:54:18.700168 7648 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e3053b529a264e8697a1df9e17eeea81 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250627 13:54:18.714418 12238 tablet_copy_client.cc:538] T 72db4bf98ab04b7189640f658d8fdcd7 P dede81a8c0164427a61532d01b6fc157: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:18.725777 12267 tablet_replica.cc:331] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854: stopping tablet replica
I20250627 13:54:18.729640 7660 catalog_manager.cc:5582] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 10 to 11, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:18.747591 12261 ts_tablet_manager.cc:1918] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20250627 13:54:18.747884 12261 log.cc:1199] T e3053b529a264e8697a1df9e17eeea81 P 9cf479bdcd0242738d2a77db4ad9f660: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal/wals/e3053b529a264e8697a1df9e17eeea81
I20250627 13:54:18.748871 7645 catalog_manager.cc:4928] TS 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933): tablet e3053b529a264e8697a1df9e17eeea81 (table rebalance_test_table_0 [id=d8bfc9268a4a4c2c92bc20d56ac73086]) successfully deleted
I20250627 13:54:18.759276 12238 tablet_bootstrap.cc:492] T 72db4bf98ab04b7189640f658d8fdcd7 P dede81a8c0164427a61532d01b6fc157: Bootstrap starting.
I20250627 13:54:18.759557 12267 raft_consensus.cc:2241] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:18.760020 12267 raft_consensus.cc:2270] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:18.764101 12267 ts_tablet_manager.cc:1905] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20250627 13:54:18.779968 8129 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.69:33101, user_credentials={real_user=slave}} blocked reactor thread for 88593us
I20250627 13:54:18.783906 9973 raft_consensus.cc:1215] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20250627 13:54:18.784559 12267 ts_tablet_manager.cc:1918] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20250627 13:54:18.784708 12267 log.cc:1199] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/wals/118d5145b23f43798e1c727f2a9e9f92
I20250627 13:54:18.785742 7647 catalog_manager.cc:4928] TS a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675): tablet 118d5145b23f43798e1c727f2a9e9f92 (table rebalance_test_table_5 [id=dd79cdfc35164ae6a649d3a77ee397c2]) successfully deleted
I20250627 13:54:18.786262 9710 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:18.800504 9973 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:18.817749 12264 tablet_copy_client.cc:323] T 570b134177314b31851951c411a9f4bf P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:18.818619 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet 570b134177314b31851951c411a9f4bf from peer 2a1282ebe92a4c9a92eed3b13f69012a ({username='slave'} at 127.0.103.69:59515)
I20250627 13:54:18.818782 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet 570b134177314b31851951c411a9f4bf from peer 2a1282ebe92a4c9a92eed3b13f69012a at {username='slave'} at 127.0.103.69:59515: session id = 2a1282ebe92a4c9a92eed3b13f69012a-570b134177314b31851951c411a9f4bf
I20250627 13:54:18.822480 8082 tablet_copy_source_session.cc:215] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:18.825392 12264 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 570b134177314b31851951c411a9f4bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:18.835000 12264 tablet_copy_client.cc:806] T 570b134177314b31851951c411a9f4bf P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:18.835242 12264 tablet_copy_client.cc:670] T 570b134177314b31851951c411a9f4bf P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:18.837704 12264 tablet_copy_client.cc:538] T 570b134177314b31851951c411a9f4bf P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:18.843129 12264 tablet_bootstrap.cc:492] T 570b134177314b31851951c411a9f4bf P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap starting.
I20250627 13:54:18.939436 8190 consensus_queue.cc:237] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
W20250627 13:54:18.958319 8126 consensus_peers.cc:487] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:18.959285 8062 consensus_queue.cc:237] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:18.965019 7926 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:18.965890 11862 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:18.966398 10102 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:18.967000 11862 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20250627 13:54:18.971290 7999 consensus_peers.cc:487] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101): Couldn't send request to peer 2a1282ebe92a4c9a92eed3b13f69012a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e2bd060a4a9e4765b45b0242ae56e489. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:18.971326 9840 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:18.972008 12180 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:18.978730 7801 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:18.979243 12180 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:18.979544 12264 tablet_bootstrap.cc:492] T 570b134177314b31851951c411a9f4bf P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:18.980000 12264 tablet_bootstrap.cc:492] T 570b134177314b31851951c411a9f4bf P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap complete.
I20250627 13:54:18.980291 12264 ts_tablet_manager.cc:1397] T 570b134177314b31851951c411a9f4bf P 2a1282ebe92a4c9a92eed3b13f69012a: Time spent bootstrapping tablet: real 0.137s user 0.053s sys 0.031s
I20250627 13:54:18.981258 12180 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:18.981859 12264 raft_consensus.cc:357] T 570b134177314b31851951c411a9f4bf P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:18.982071 12264 raft_consensus.cc:738] T 570b134177314b31851951c411a9f4bf P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a1282ebe92a4c9a92eed3b13f69012a, State: Initialized, Role: LEARNER
I20250627 13:54:18.982280 12264 consensus_queue.cc:260] T 570b134177314b31851951c411a9f4bf P 2a1282ebe92a4c9a92eed3b13f69012a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:18.982208 9840 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:18.982944 12264 ts_tablet_manager.cc:1428] T 570b134177314b31851951c411a9f4bf P 2a1282ebe92a4c9a92eed3b13f69012a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:54:18.986819 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session 2a1282ebe92a4c9a92eed3b13f69012a-570b134177314b31851951c411a9f4bf received from {username='slave'} at 127.0.103.69:59515
I20250627 13:54:18.986960 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session 2a1282ebe92a4c9a92eed3b13f69012a-570b134177314b31851951c411a9f4bf on tablet 570b134177314b31851951c411a9f4bf with peer 2a1282ebe92a4c9a92eed3b13f69012a
I20250627 13:54:18.999161 7660 catalog_manager.cc:5582] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 6 to 7, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:19.018487 7800 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:19.019348 11862 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:19.026574 10102 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:19.029254 7926 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:19.029363 7660 catalog_manager.cc:5582] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 8 to 9, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:19.053851 9710 raft_consensus.cc:1215] T 570b134177314b31851951c411a9f4bf P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250627 13:54:19.059182 12238 tablet_bootstrap.cc:492] T 72db4bf98ab04b7189640f658d8fdcd7 P dede81a8c0164427a61532d01b6fc157: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:19.059698 12238 tablet_bootstrap.cc:492] T 72db4bf98ab04b7189640f658d8fdcd7 P dede81a8c0164427a61532d01b6fc157: Bootstrap complete.
I20250627 13:54:19.060007 12238 ts_tablet_manager.cc:1397] T 72db4bf98ab04b7189640f658d8fdcd7 P dede81a8c0164427a61532d01b6fc157: Time spent bootstrapping tablet: real 0.301s user 0.055s sys 0.009s
I20250627 13:54:19.060983 12238 raft_consensus.cc:357] T 72db4bf98ab04b7189640f658d8fdcd7 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:19.061259 12238 raft_consensus.cc:738] T 72db4bf98ab04b7189640f658d8fdcd7 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dede81a8c0164427a61532d01b6fc157, State: Initialized, Role: LEARNER
I20250627 13:54:19.061520 12238 consensus_queue.cc:260] T 72db4bf98ab04b7189640f658d8fdcd7 P dede81a8c0164427a61532d01b6fc157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:19.062363 12238 ts_tablet_manager.cc:1428] T 72db4bf98ab04b7189640f658d8fdcd7 P dede81a8c0164427a61532d01b6fc157: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250627 13:54:19.064980 7953 tablet_copy_service.cc:342] P a41a89ae8ed743edb2b5cf4490d8c854: Request end of tablet copy session dede81a8c0164427a61532d01b6fc157-72db4bf98ab04b7189640f658d8fdcd7 received from {username='slave'} at 127.0.103.72:34895
I20250627 13:54:19.065147 7953 tablet_copy_service.cc:434] P a41a89ae8ed743edb2b5cf4490d8c854: ending tablet copy session dede81a8c0164427a61532d01b6fc157-72db4bf98ab04b7189640f658d8fdcd7 on tablet 72db4bf98ab04b7189640f658d8fdcd7 with peer dede81a8c0164427a61532d01b6fc157
I20250627 13:54:19.137337 7926 consensus_queue.cc:237] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:19.141206 8062 consensus_queue.cc:237] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:19.151386 7800 raft_consensus.cc:1273] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:19.151351 9710 raft_consensus.cc:1273] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:19.152124 11961 consensus_queue.cc:1035] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:19.152503 12179 consensus_queue.cc:1035] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:19.153982 7927 raft_consensus.cc:1273] T eed3236c4fe8455cb9527b9f4b754e81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:19.155346 11961 consensus_queue.cc:1035] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:19.157193 9840 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20250627 13:54:19.158931 7868 consensus_peers.cc:487] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 -> Peer dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213): Couldn't send request to peer dede81a8c0164427a61532d01b6fc157. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03640a5124d2405faf4a26d0e22e1ffc. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:19.159152 12115 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:19.161741 8190 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:19.164680 11959 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:19.176323 11959 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:19.178401 9842 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:19.178786 11961 raft_consensus.cc:2953] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:19.179425 8062 consensus_queue.cc:237] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:19.185392 7926 raft_consensus.cc:2953] T eed3236c4fe8455cb9527b9f4b754e81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:19.187228 7800 raft_consensus.cc:2953] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:19.192323 10102 raft_consensus.cc:1215] T 72db4bf98ab04b7189640f658d8fdcd7 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250627 13:54:19.194495 8190 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:19.198078 7660 catalog_manager.cc:5582] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 5 to 6. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:19.204087 7657 catalog_manager.cc:5582] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 9 to 10, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:19.206794 9710 raft_consensus.cc:2953] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:19.214900 8061 consensus_queue.cc:237] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:19.217339 9710 raft_consensus.cc:1273] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20250627 13:54:19.218050 7645 catalog_manager.cc:5112] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet eed3236c4fe8455cb9527b9f4b754e81 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:19.218080 12179 consensus_queue.cc:1035] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:19.221004 7800 raft_consensus.cc:1273] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:19.220948 12179 raft_consensus.cc:2953] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:19.222317 9710 raft_consensus.cc:2953] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
W20250627 13:54:19.223832 7999 consensus_peers.cc:487] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 6686c6c365bf4f4582058890161676fe (127.0.103.71:37859): Couldn't send request to peer 6686c6c365bf4f4582058890161676fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9196497cb11f4ccbbd62695253d4c8c9. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:19.224006 12180 consensus_queue.cc:1035] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:19.227658 7800 raft_consensus.cc:2953] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:19.229475 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet eed3236c4fe8455cb9527b9f4b754e81 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:19.229928 9842 raft_consensus.cc:1273] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:19.230520 12179 consensus_queue.cc:1035] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:19.230831 8190 raft_consensus.cc:1273] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:19.232069 12180 consensus_queue.cc:1035] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:19.233439 7653 catalog_manager.cc:5582] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 6 to 7, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:19.237915 7911 tablet_service.cc:1515] Processing DeleteTablet for tablet eed3236c4fe8455cb9527b9f4b754e81 with delete_type TABLET_DATA_TOMBSTONED (TS a41a89ae8ed743edb2b5cf4490d8c854 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59302
I20250627 13:54:19.238299 12267 tablet_replica.cc:331] T eed3236c4fe8455cb9527b9f4b754e81 P a41a89ae8ed743edb2b5cf4490d8c854: stopping tablet replica
I20250627 13:54:19.238559 12267 raft_consensus.cc:2241] T eed3236c4fe8455cb9527b9f4b754e81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:19.238925 12267 raft_consensus.cc:2270] T eed3236c4fe8455cb9527b9f4b754e81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:19.240917 12267 ts_tablet_manager.cc:1905] T eed3236c4fe8455cb9527b9f4b754e81 P a41a89ae8ed743edb2b5cf4490d8c854: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:19.251973 8062 consensus_queue.cc:237] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
W20250627 13:54:19.256361 7996 consensus_peers.cc:487] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eed3236c4fe8455cb9527b9f4b754e81. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:19.257364 9710 raft_consensus.cc:1273] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:19.257882 11961 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:19.258388 9841 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:19.265853 12180 consensus_queue.cc:1035] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:19.266481 7658 catalog_manager.cc:5582] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b reported cstate change: config changed from index 6 to 7, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:19.271934 7645 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet eed3236c4fe8455cb9527b9f4b754e81 with cas_config_opid_index 6: aborting the task: latest config opid_index 7; task opid_index 6
I20250627 13:54:19.276948 7801 raft_consensus.cc:1273] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:19.277513 12339 ts_tablet_manager.cc:927] T e3053b529a264e8697a1df9e17eeea81 P 956ff058e6d64579a1963aabdb6d466b: Initiating tablet copy from peer e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:54:19.277974 12179 consensus_queue.cc:1035] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:19.283836 12267 ts_tablet_manager.cc:1918] T eed3236c4fe8455cb9527b9f4b754e81 P a41a89ae8ed743edb2b5cf4490d8c854: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20250627 13:54:19.284979 11961 raft_consensus.cc:2953] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:19.287457 12267 log.cc:1199] T eed3236c4fe8455cb9527b9f4b754e81 P a41a89ae8ed743edb2b5cf4490d8c854: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/wals/eed3236c4fe8455cb9527b9f4b754e81
I20250627 13:54:19.288674 7647 catalog_manager.cc:4928] TS a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675): tablet eed3236c4fe8455cb9527b9f4b754e81 (table rebalance_test_table_9 [id=19e5686b50fe4187a733aa6413ef2ce0]) successfully deleted
I20250627 13:54:19.290009 7645 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet eed3236c4fe8455cb9527b9f4b754e81 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250627 13:54:19.289749 7800 raft_consensus.cc:2953] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:19.292155 7653 catalog_manager.cc:5582] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 7 to 8, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:19.313877 12339 tablet_copy_client.cc:323] T e3053b529a264e8697a1df9e17eeea81 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.68:46139
I20250627 13:54:19.314716 8213 tablet_copy_service.cc:140] P e4c831df4da841929d13d6c783099b0e: Received BeginTabletCopySession request for tablet e3053b529a264e8697a1df9e17eeea81 from peer 956ff058e6d64579a1963aabdb6d466b ({username='slave'} at 127.0.103.70:33381)
I20250627 13:54:19.314945 8213 tablet_copy_service.cc:161] P e4c831df4da841929d13d6c783099b0e: Beginning new tablet copy session on tablet e3053b529a264e8697a1df9e17eeea81 from peer 956ff058e6d64579a1963aabdb6d466b at {username='slave'} at 127.0.103.70:33381: session id = 956ff058e6d64579a1963aabdb6d466b-e3053b529a264e8697a1df9e17eeea81
I20250627 13:54:19.315742 9710 raft_consensus.cc:2953] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:19.318915 8213 tablet_copy_source_session.cc:215] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:19.319640 8190 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:19.333765 12339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3053b529a264e8697a1df9e17eeea81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:19.341639 12339 tablet_copy_client.cc:806] T e3053b529a264e8697a1df9e17eeea81 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:19.341903 12339 tablet_copy_client.cc:670] T e3053b529a264e8697a1df9e17eeea81 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:19.360144 12339 tablet_copy_client.cc:538] T e3053b529a264e8697a1df9e17eeea81 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:19.364712 12339 tablet_bootstrap.cc:492] T e3053b529a264e8697a1df9e17eeea81 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap starting.
I20250627 13:54:19.365581 8062 consensus_queue.cc:237] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:19.370774 8063 consensus_queue.cc:237] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
W20250627 13:54:19.380818 7996 consensus_peers.cc:487] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5184c3f8e465472e9a98c14f8e703415. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:19.381858 9973 raft_consensus.cc:1273] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:19.382397 12340 consensus_queue.cc:1035] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:19.385538 7800 raft_consensus.cc:1273] T 5184c3f8e465472e9a98c14f8e703415 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:19.386226 12340 consensus_queue.cc:1035] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:19.398649 12340 raft_consensus.cc:2953] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:19.405930 9973 raft_consensus.cc:2953] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:19.412295 7653 catalog_manager.cc:5582] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 6 to 7, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:19.424830 7801 raft_consensus.cc:2953] T 5184c3f8e465472e9a98c14f8e703415 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:19.441222 12369 ts_tablet_manager.cc:927] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: Initiating tablet copy from peer e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:54:19.444134 7926 raft_consensus.cc:1273] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:19.445261 8190 raft_consensus.cc:1273] T 83cfc975906745a09c5e1dcfac812ae2 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250627 13:54:19.448424 12340 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250627 13:54:19.450642 12180 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250627 13:54:19.453418 7998 consensus_peers.cc:487] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213): Couldn't send request to peer dede81a8c0164427a61532d01b6fc157. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83cfc975906745a09c5e1dcfac812ae2. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:19.460047 12340 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:19.464284 12369 tablet_copy_client.cc:323] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.68:46139
I20250627 13:54:19.464819 8190 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:19.466567 8213 tablet_copy_service.cc:140] P e4c831df4da841929d13d6c783099b0e: Received BeginTabletCopySession request for tablet 14dba72d45694defb83f41c1572722ca from peer 956ff058e6d64579a1963aabdb6d466b ({username='slave'} at 127.0.103.70:33381)
I20250627 13:54:19.466722 8213 tablet_copy_service.cc:161] P e4c831df4da841929d13d6c783099b0e: Beginning new tablet copy session on tablet 14dba72d45694defb83f41c1572722ca from peer 956ff058e6d64579a1963aabdb6d466b at {username='slave'} at 127.0.103.70:33381: session id = 956ff058e6d64579a1963aabdb6d466b-14dba72d45694defb83f41c1572722ca
I20250627 13:54:19.469467 8213 tablet_copy_source_session.cc:215] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:19.471621 7653 catalog_manager.cc:5582] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:19.474907 7926 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:19.475577 7800 consensus_queue.cc:237] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:19.479378 12369 ts_tablet_manager.cc:1905] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250627 13:54:19.487776 12369 ts_tablet_manager.cc:1918] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20250627 13:54:19.488256 10102 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:19.489606 12082 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:19.490170 12369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14dba72d45694defb83f41c1572722ca. 1 dirs total, 0 dirs full, 0 dirs failed
W20250627 13:54:19.490728 7738 consensus_peers.cc:487] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 -> Peer 6686c6c365bf4f4582058890161676fe (127.0.103.71:37859): Couldn't send request to peer 6686c6c365bf4f4582058890161676fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 897206e017ff4305a8cc9092e30f46b6. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:19.491148 8190 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:19.492137 12082 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:19.495287 12369 tablet_copy_client.cc:806] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:19.495558 12369 tablet_copy_client.cc:670] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:19.497534 12369 tablet_copy_client.cc:538] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:19.501988 12369 tablet_bootstrap.cc:492] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: Bootstrap starting.
I20250627 13:54:19.529639 7801 consensus_queue.cc:237] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:19.537976 12082 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:19.544961 8190 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:19.550607 12238 ts_tablet_manager.cc:927] T 03640a5124d2405faf4a26d0e22e1ffc P dede81a8c0164427a61532d01b6fc157: Initiating tablet copy from peer a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:54:19.551611 9973 raft_consensus.cc:1273] T ff83fdbab2b9417eac50495c54f0da54 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:19.551642 7926 raft_consensus.cc:1273] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20250627 13:54:19.551621 7737 consensus_peers.cc:487] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 -> Peer dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213): Couldn't send request to peer dede81a8c0164427a61532d01b6fc157. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff83fdbab2b9417eac50495c54f0da54. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:19.552207 11920 consensus_queue.cc:1035] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:19.552546 12386 consensus_queue.cc:1035] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:19.552846 12238 tablet_copy_client.cc:323] T 03640a5124d2405faf4a26d0e22e1ffc P dede81a8c0164427a61532d01b6fc157: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:43675
I20250627 13:54:19.552902 10104 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:19.553846 7953 tablet_copy_service.cc:140] P a41a89ae8ed743edb2b5cf4490d8c854: Received BeginTabletCopySession request for tablet 03640a5124d2405faf4a26d0e22e1ffc from peer dede81a8c0164427a61532d01b6fc157 ({username='slave'} at 127.0.103.72:34895)
I20250627 13:54:19.554011 7953 tablet_copy_service.cc:161] P a41a89ae8ed743edb2b5cf4490d8c854: Beginning new tablet copy session on tablet 03640a5124d2405faf4a26d0e22e1ffc from peer dede81a8c0164427a61532d01b6fc157 at {username='slave'} at 127.0.103.72:34895: session id = dede81a8c0164427a61532d01b6fc157-03640a5124d2405faf4a26d0e22e1ffc
I20250627 13:54:19.556941 7658 catalog_manager.cc:5582] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 6 to 7, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:19.558111 7953 tablet_copy_source_session.cc:215] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:19.559543 12238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03640a5124d2405faf4a26d0e22e1ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:19.576501 12238 tablet_copy_client.cc:806] T 03640a5124d2405faf4a26d0e22e1ffc P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:19.576910 12238 tablet_copy_client.cc:670] T 03640a5124d2405faf4a26d0e22e1ffc P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:19.579073 12238 tablet_copy_client.cc:538] T 03640a5124d2405faf4a26d0e22e1ffc P dede81a8c0164427a61532d01b6fc157: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:19.584793 12238 tablet_bootstrap.cc:492] T 03640a5124d2405faf4a26d0e22e1ffc P dede81a8c0164427a61532d01b6fc157: Bootstrap starting.
I20250627 13:54:19.589006 12339 tablet_bootstrap.cc:492] T e3053b529a264e8697a1df9e17eeea81 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:19.589422 12339 tablet_bootstrap.cc:492] T e3053b529a264e8697a1df9e17eeea81 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap complete.
I20250627 13:54:19.594777 12400 ts_tablet_manager.cc:927] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:19.603013 12340 raft_consensus.cc:1062] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a to VOTER
I20250627 13:54:19.603751 12339 ts_tablet_manager.cc:1397] T e3053b529a264e8697a1df9e17eeea81 P 956ff058e6d64579a1963aabdb6d466b: Time spent bootstrapping tablet: real 0.240s user 0.061s sys 0.009s
I20250627 13:54:19.603984 12340 consensus_queue.cc:237] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:19.604560 12339 raft_consensus.cc:357] T e3053b529a264e8697a1df9e17eeea81 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:19.604780 12339 raft_consensus.cc:738] T e3053b529a264e8697a1df9e17eeea81 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 956ff058e6d64579a1963aabdb6d466b, State: Initialized, Role: LEARNER
I20250627 13:54:19.605515 12339 consensus_queue.cc:260] T e3053b529a264e8697a1df9e17eeea81 P 956ff058e6d64579a1963aabdb6d466b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:19.607728 9840 raft_consensus.cc:1273] T 570b134177314b31851951c411a9f4bf P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:19.608739 12179 consensus_queue.cc:1035] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:19.608800 8190 raft_consensus.cc:1273] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:19.609345 12179 consensus_queue.cc:1035] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:19.610013 12339 ts_tablet_manager.cc:1428] T e3053b529a264e8697a1df9e17eeea81 P 956ff058e6d64579a1963aabdb6d466b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250627 13:54:19.610718 9710 raft_consensus.cc:1273] T 570b134177314b31851951c411a9f4bf P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:19.611093 8213 tablet_copy_service.cc:342] P e4c831df4da841929d13d6c783099b0e: Request end of tablet copy session 956ff058e6d64579a1963aabdb6d466b-e3053b529a264e8697a1df9e17eeea81 received from {username='slave'} at 127.0.103.70:33381
I20250627 13:54:19.611217 8213 tablet_copy_service.cc:434] P e4c831df4da841929d13d6c783099b0e: ending tablet copy session 956ff058e6d64579a1963aabdb6d466b-e3053b529a264e8697a1df9e17eeea81 on tablet e3053b529a264e8697a1df9e17eeea81 with peer 956ff058e6d64579a1963aabdb6d466b
I20250627 13:54:19.611990 12179 consensus_queue.cc:1035] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:19.581037 12385 raft_consensus.cc:2953] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:19.620910 11959 raft_consensus.cc:1062] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854: attempting to promote NON_VOTER dede81a8c0164427a61532d01b6fc157 to VOTER
I20250627 13:54:19.621829 11959 consensus_queue.cc:237] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } }
I20250627 13:54:19.627194 12179 raft_consensus.cc:2953] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:19.633085 9840 raft_consensus.cc:2953] T 570b134177314b31851951c411a9f4bf P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:19.634577 9710 raft_consensus.cc:2953] T 570b134177314b31851951c411a9f4bf P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:19.638769 9709 raft_consensus.cc:1273] T 72db4bf98ab04b7189640f658d8fdcd7 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:19.639678 11959 consensus_queue.cc:1035] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:19.641337 7658 catalog_manager.cc:5582] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 7 to 8, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:19.646317 10104 raft_consensus.cc:1273] T 72db4bf98ab04b7189640f658d8fdcd7 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:19.648260 7801 raft_consensus.cc:1273] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:19.648981 11959 consensus_queue.cc:1035] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:19.650118 11959 consensus_queue.cc:1035] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:19.651530 8062 consensus_queue.cc:237] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:19.655742 9840 raft_consensus.cc:1273] T 570b134177314b31851951c411a9f4bf P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:19.656351 11961 consensus_queue.cc:1035] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:19.658201 12400 tablet_copy_client.cc:323] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:19.658690 12340 raft_consensus.cc:2953] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:19.659405 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet e2bd060a4a9e4765b45b0242ae56e489 from peer 2a1282ebe92a4c9a92eed3b13f69012a ({username='slave'} at 127.0.103.69:59515)
I20250627 13:54:19.659579 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet e2bd060a4a9e4765b45b0242ae56e489 from peer 2a1282ebe92a4c9a92eed3b13f69012a at {username='slave'} at 127.0.103.69:59515: session id = 2a1282ebe92a4c9a92eed3b13f69012a-e2bd060a4a9e4765b45b0242ae56e489
I20250627 13:54:19.661448 9711 raft_consensus.cc:1273] T 570b134177314b31851951c411a9f4bf P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:19.662463 11961 consensus_queue.cc:1035] T 570b134177314b31851951c411a9f4bf P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:19.664297 8082 tablet_copy_source_session.cc:215] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:19.664014 9711 raft_consensus.cc:2953] T 570b134177314b31851951c411a9f4bf P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:19.668673 9842 raft_consensus.cc:2953] T 570b134177314b31851951c411a9f4bf P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:19.672374 7653 catalog_manager.cc:5582] T 570b134177314b31851951c411a9f4bf P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: config changed from index 8 to 9, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:19.676337 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 570b134177314b31851951c411a9f4bf with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:19.678704 8171 tablet_service.cc:1515] Processing DeleteTablet for tablet 570b134177314b31851951c411a9f4bf with delete_type TABLET_DATA_TOMBSTONED (TS e4c831df4da841929d13d6c783099b0e not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:48616
I20250627 13:54:19.678907 7927 raft_consensus.cc:2953] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:19.686256 12425 tablet_replica.cc:331] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e: stopping tablet replica
I20250627 13:54:19.686973 12425 raft_consensus.cc:2241] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:19.688946 12425 pending_rounds.cc:70] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e: Trying to abort 1 pending ops.
I20250627 13:54:19.689390 12425 pending_rounds.cc:77] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7172228954535751680 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "570b134177314b31851951c411a9f4bf" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } } }
I20250627 13:54:19.693845 12425 raft_consensus.cc:2915] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20250627 13:54:19.694172 12425 raft_consensus.cc:2887] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250627 13:54:19.694422 12425 raft_consensus.cc:2270] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:19.695600 12115 raft_consensus.cc:2953] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:19.696461 7800 raft_consensus.cc:2953] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:19.696619 9708 raft_consensus.cc:2953] T 72db4bf98ab04b7189640f658d8fdcd7 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:19.702551 7660 catalog_manager.cc:5582] T 72db4bf98ab04b7189640f658d8fdcd7 P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: config changed from index 7 to 8, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:19.707646 7653 catalog_manager.cc:5582] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index 6 to 7, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:19.724231 9840 raft_consensus.cc:1215] T e3053b529a264e8697a1df9e17eeea81 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20250627 13:54:19.735304 12400 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2bd060a4a9e4765b45b0242ae56e489. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:19.738401 9973 raft_consensus.cc:2953] T ff83fdbab2b9417eac50495c54f0da54 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:19.740075 12427 ts_tablet_manager.cc:927] T 9196497cb11f4ccbbd62695253d4c8c9 P 6686c6c365bf4f4582058890161676fe: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:19.753222 12369 tablet_bootstrap.cc:492] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:19.753680 12369 tablet_bootstrap.cc:492] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: Bootstrap complete.
I20250627 13:54:19.754630 12425 ts_tablet_manager.cc:1905] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:19.754112 10104 raft_consensus.cc:2953] T 72db4bf98ab04b7189640f658d8fdcd7 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:19.761435 12425 ts_tablet_manager.cc:1918] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20250627 13:54:19.761606 12425 log.cc:1199] T 570b134177314b31851951c411a9f4bf P e4c831df4da841929d13d6c783099b0e: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal/wals/570b134177314b31851951c411a9f4bf
I20250627 13:54:19.763387 7648 catalog_manager.cc:4928] TS e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): tablet 570b134177314b31851951c411a9f4bf (table rebalance_test_table_7 [id=46b91111142d4e388d01cd06d3d41940]) successfully deleted
I20250627 13:54:19.765237 12369 ts_tablet_manager.cc:1397] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: Time spent bootstrapping tablet: real 0.263s user 0.047s sys 0.014s
I20250627 13:54:19.766233 12369 raft_consensus.cc:357] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:19.766450 12369 raft_consensus.cc:738] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 956ff058e6d64579a1963aabdb6d466b, State: Initialized, Role: LEARNER
I20250627 13:54:19.766667 12369 consensus_queue.cc:260] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:19.768085 7926 consensus_queue.cc:237] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } }
I20250627 13:54:19.771615 9711 raft_consensus.cc:1273] T 72db4bf98ab04b7189640f658d8fdcd7 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:19.771735 12369 ts_tablet_manager.cc:1428] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250627 13:54:19.772420 8213 tablet_copy_service.cc:342] P e4c831df4da841929d13d6c783099b0e: Request end of tablet copy session 956ff058e6d64579a1963aabdb6d466b-14dba72d45694defb83f41c1572722ca received from {username='slave'} at 127.0.103.70:33381
I20250627 13:54:19.772544 8213 tablet_copy_service.cc:434] P e4c831df4da841929d13d6c783099b0e: ending tablet copy session 956ff058e6d64579a1963aabdb6d466b-14dba72d45694defb83f41c1572722ca on tablet 14dba72d45694defb83f41c1572722ca with peer 956ff058e6d64579a1963aabdb6d466b
I20250627 13:54:19.773059 11959 consensus_queue.cc:1035] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:19.777416 9711 raft_consensus.cc:2953] T 72db4bf98ab04b7189640f658d8fdcd7 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:19.782642 12416 raft_consensus.cc:2953] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:19.786252 7658 catalog_manager.cc:5582] T 72db4bf98ab04b7189640f658d8fdcd7 P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: config changed from index 8 to 9, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:19.787542 7647 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 72db4bf98ab04b7189640f658d8fdcd7 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:19.791720 12427 tablet_copy_client.cc:323] T 9196497cb11f4ccbbd62695253d4c8c9 P 6686c6c365bf4f4582058890161676fe: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:19.792869 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet 9196497cb11f4ccbbd62695253d4c8c9 from peer 6686c6c365bf4f4582058890161676fe ({username='slave'} at 127.0.103.71:59981)
I20250627 13:54:19.793016 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet 9196497cb11f4ccbbd62695253d4c8c9 from peer 6686c6c365bf4f4582058890161676fe at {username='slave'} at 127.0.103.71:59981: session id = 6686c6c365bf4f4582058890161676fe-9196497cb11f4ccbbd62695253d4c8c9
I20250627 13:54:19.794938 10102 raft_consensus.cc:1273] T 72db4bf98ab04b7189640f658d8fdcd7 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:19.795519 12416 consensus_queue.cc:1035] T 72db4bf98ab04b7189640f658d8fdcd7 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:19.795774 8082 tablet_copy_source_session.cc:215] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:19.797065 12427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9196497cb11f4ccbbd62695253d4c8c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:19.796756 10102 raft_consensus.cc:2953] T 72db4bf98ab04b7189640f658d8fdcd7 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:19.797693 7782 tablet_service.cc:1515] Processing DeleteTablet for tablet 72db4bf98ab04b7189640f658d8fdcd7 with delete_type TABLET_DATA_TOMBSTONED (TS 83574c1bf8ea45adaff939158f93c096 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:45826
I20250627 13:54:19.802707 12400 tablet_copy_client.cc:806] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:19.806674 12400 tablet_copy_client.cc:670] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:19.811158 12400 tablet_copy_client.cc:538] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:19.820765 12427 tablet_copy_client.cc:806] T 9196497cb11f4ccbbd62695253d4c8c9 P 6686c6c365bf4f4582058890161676fe: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:19.820668 7800 consensus_queue.cc:237] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:19.821287 12427 tablet_copy_client.cc:670] T 9196497cb11f4ccbbd62695253d4c8c9 P 6686c6c365bf4f4582058890161676fe: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:19.824249 12427 tablet_copy_client.cc:538] T 9196497cb11f4ccbbd62695253d4c8c9 P 6686c6c365bf4f4582058890161676fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:19.825064 12400 tablet_bootstrap.cc:492] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap starting.
I20250627 13:54:19.829878 12432 tablet_replica.cc:331] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096: stopping tablet replica
I20250627 13:54:19.830276 12432 raft_consensus.cc:2241] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:19.830677 12432 raft_consensus.cc:2270] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:19.834667 9711 raft_consensus.cc:1273] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:19.817612 12238 tablet_bootstrap.cc:492] T 03640a5124d2405faf4a26d0e22e1ffc P dede81a8c0164427a61532d01b6fc157: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:19.835335 12385 consensus_queue.cc:1035] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:19.835953 12238 tablet_bootstrap.cc:492] T 03640a5124d2405faf4a26d0e22e1ffc P dede81a8c0164427a61532d01b6fc157: Bootstrap complete.
I20250627 13:54:19.836817 12432 ts_tablet_manager.cc:1905] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:19.837204 12238 ts_tablet_manager.cc:1397] T 03640a5124d2405faf4a26d0e22e1ffc P dede81a8c0164427a61532d01b6fc157: Time spent bootstrapping tablet: real 0.253s user 0.049s sys 0.007s
I20250627 13:54:19.838161 12238 raft_consensus.cc:357] T 03640a5124d2405faf4a26d0e22e1ffc P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:19.838968 7926 raft_consensus.cc:1273] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:19.839571 11920 consensus_queue.cc:1035] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:19.839648 12238 raft_consensus.cc:738] T 03640a5124d2405faf4a26d0e22e1ffc P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dede81a8c0164427a61532d01b6fc157, State: Initialized, Role: LEARNER
I20250627 13:54:19.840016 12238 consensus_queue.cc:260] T 03640a5124d2405faf4a26d0e22e1ffc P dede81a8c0164427a61532d01b6fc157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:19.842007 12238 ts_tablet_manager.cc:1428] T 03640a5124d2405faf4a26d0e22e1ffc P dede81a8c0164427a61532d01b6fc157: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250627 13:54:19.846457 7953 tablet_copy_service.cc:342] P a41a89ae8ed743edb2b5cf4490d8c854: Request end of tablet copy session dede81a8c0164427a61532d01b6fc157-03640a5124d2405faf4a26d0e22e1ffc received from {username='slave'} at 127.0.103.72:34895
I20250627 13:54:19.846630 7953 tablet_copy_service.cc:434] P a41a89ae8ed743edb2b5cf4490d8c854: ending tablet copy session dede81a8c0164427a61532d01b6fc157-03640a5124d2405faf4a26d0e22e1ffc on tablet 03640a5124d2405faf4a26d0e22e1ffc with peer dede81a8c0164427a61532d01b6fc157
I20250627 13:54:19.854126 12386 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:19.854775 9709 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:19.859762 12427 tablet_bootstrap.cc:492] T 9196497cb11f4ccbbd62695253d4c8c9 P 6686c6c365bf4f4582058890161676fe: Bootstrap starting.
I20250627 13:54:19.861387 7658 catalog_manager.cc:5582] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: config changed from index 6 to 7, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:19.870306 12369 ts_tablet_manager.cc:927] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:19.869359 7926 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:19.870308 9840 raft_consensus.cc:1215] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20250627 13:54:19.878237 12369 tablet_copy_client.cc:323] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:19.878860 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet eed3236c4fe8455cb9527b9f4b754e81 from peer 956ff058e6d64579a1963aabdb6d466b ({username='slave'} at 127.0.103.70:59981)
I20250627 13:54:19.879036 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet eed3236c4fe8455cb9527b9f4b754e81 from peer 956ff058e6d64579a1963aabdb6d466b at {username='slave'} at 127.0.103.70:59981: session id = 956ff058e6d64579a1963aabdb6d466b-eed3236c4fe8455cb9527b9f4b754e81
I20250627 13:54:19.880378 12432 ts_tablet_manager.cc:1918] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20250627 13:54:19.880925 12432 log.cc:1199] T 72db4bf98ab04b7189640f658d8fdcd7 P 83574c1bf8ea45adaff939158f93c096: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/wals/72db4bf98ab04b7189640f658d8fdcd7
I20250627 13:54:19.881744 7645 catalog_manager.cc:4928] TS 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467): tablet 72db4bf98ab04b7189640f658d8fdcd7 (table rebalance_test_table_8 [id=b84f3fe3fd7c4f01902547d2db982f90]) successfully deleted
I20250627 13:54:19.882640 8082 tablet_copy_source_session.cc:215] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:19.884131 12339 ts_tablet_manager.cc:927] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:19.884876 12369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eed3236c4fe8455cb9527b9f4b754e81. 1 dirs total, 0 dirs full, 0 dirs failed
W20250627 13:54:19.889258 7735 consensus_peers.cc:487] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1dc569a7b12d40058d073f7b81b2601c. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:19.894930 8062 consensus_queue.cc:237] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:19.897953 7927 raft_consensus.cc:1273] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:19.898810 11961 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250627 13:54:19.899549 7996 consensus_peers.cc:487] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83cfc975906745a09c5e1dcfac812ae2. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:19.900144 12339 tablet_copy_client.cc:323] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:19.900802 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet 5184c3f8e465472e9a98c14f8e703415 from peer 956ff058e6d64579a1963aabdb6d466b ({username='slave'} at 127.0.103.70:59981)
I20250627 13:54:19.900992 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet 5184c3f8e465472e9a98c14f8e703415 from peer 956ff058e6d64579a1963aabdb6d466b at {username='slave'} at 127.0.103.70:59981: session id = 956ff058e6d64579a1963aabdb6d466b-5184c3f8e465472e9a98c14f8e703415
I20250627 13:54:19.901113 8190 raft_consensus.cc:1273] T 83cfc975906745a09c5e1dcfac812ae2 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250627 13:54:19.901710 12340 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250627 13:54:19.902215 12340 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:19.902702 7926 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:19.904403 8082 tablet_copy_source_session.cc:215] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:19.905521 12339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5184c3f8e465472e9a98c14f8e703415. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:19.906682 8190 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:19.909297 12369 tablet_copy_client.cc:806] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:19.909525 12369 tablet_copy_client.cc:670] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:19.913843 12369 tablet_copy_client.cc:538] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:19.914395 7658 catalog_manager.cc:5582] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index 4 to 5, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:19.917132 12369 tablet_bootstrap.cc:492] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap starting.
I20250627 13:54:19.922437 12339 tablet_copy_client.cc:806] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:19.929431 12339 tablet_copy_client.cc:670] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 1 WAL segments...
W20250627 13:54:19.930819 7998 consensus_peers.cc:487] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213): Couldn't send request to peer dede81a8c0164427a61532d01b6fc157. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83cfc975906745a09c5e1dcfac812ae2. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:19.931577 12339 tablet_copy_client.cc:538] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:19.935057 12339 tablet_bootstrap.cc:492] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap starting.
I20250627 13:54:19.945688 12400 tablet_bootstrap.cc:492] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:19.946121 12400 tablet_bootstrap.cc:492] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap complete.
I20250627 13:54:19.946435 12400 ts_tablet_manager.cc:1397] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a: Time spent bootstrapping tablet: real 0.121s user 0.045s sys 0.004s
I20250627 13:54:19.947564 12400 raft_consensus.cc:357] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:19.947782 12400 raft_consensus.cc:738] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a1282ebe92a4c9a92eed3b13f69012a, State: Initialized, Role: LEARNER
I20250627 13:54:19.947966 12400 consensus_queue.cc:260] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:19.952150 12400 ts_tablet_manager.cc:1428] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250627 13:54:19.955507 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session 2a1282ebe92a4c9a92eed3b13f69012a-e2bd060a4a9e4765b45b0242ae56e489 received from {username='slave'} at 127.0.103.69:59515
I20250627 13:54:19.955679 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session 2a1282ebe92a4c9a92eed3b13f69012a-e2bd060a4a9e4765b45b0242ae56e489 on tablet e2bd060a4a9e4765b45b0242ae56e489 with peer 2a1282ebe92a4c9a92eed3b13f69012a
I20250627 13:54:19.986356 12461 ts_tablet_manager.cc:927] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe: Initiating tablet copy from peer 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:54:19.987974 12461 tablet_copy_client.cc:323] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:37467
I20250627 13:54:19.988685 7822 tablet_copy_service.cc:140] P 83574c1bf8ea45adaff939158f93c096: Received BeginTabletCopySession request for tablet 897206e017ff4305a8cc9092e30f46b6 from peer 6686c6c365bf4f4582058890161676fe ({username='slave'} at 127.0.103.71:53529)
I20250627 13:54:19.988933 7822 tablet_copy_service.cc:161] P 83574c1bf8ea45adaff939158f93c096: Beginning new tablet copy session on tablet 897206e017ff4305a8cc9092e30f46b6 from peer 6686c6c365bf4f4582058890161676fe at {username='slave'} at 127.0.103.71:53529: session id = 6686c6c365bf4f4582058890161676fe-897206e017ff4305a8cc9092e30f46b6
I20250627 13:54:19.989547 12238 ts_tablet_manager.cc:927] T ff83fdbab2b9417eac50495c54f0da54 P dede81a8c0164427a61532d01b6fc157: Initiating tablet copy from peer 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:54:19.991919 7822 tablet_copy_source_session.cc:215] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:19.993935 12238 tablet_copy_client.cc:323] T ff83fdbab2b9417eac50495c54f0da54 P dede81a8c0164427a61532d01b6fc157: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:37467
I20250627 13:54:19.994539 7822 tablet_copy_service.cc:140] P 83574c1bf8ea45adaff939158f93c096: Received BeginTabletCopySession request for tablet ff83fdbab2b9417eac50495c54f0da54 from peer dede81a8c0164427a61532d01b6fc157 ({username='slave'} at 127.0.103.72:38213)
I20250627 13:54:19.994726 7822 tablet_copy_service.cc:161] P 83574c1bf8ea45adaff939158f93c096: Beginning new tablet copy session on tablet ff83fdbab2b9417eac50495c54f0da54 from peer dede81a8c0164427a61532d01b6fc157 at {username='slave'} at 127.0.103.72:38213: session id = dede81a8c0164427a61532d01b6fc157-ff83fdbab2b9417eac50495c54f0da54
I20250627 13:54:19.997467 7822 tablet_copy_source_session.cc:215] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:19.998421 12461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 897206e017ff4305a8cc9092e30f46b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:19.998637 12238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff83fdbab2b9417eac50495c54f0da54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:20.001691 12369 tablet_bootstrap.cc:492] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:20.002130 12369 tablet_bootstrap.cc:492] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap complete.
I20250627 13:54:20.002372 12369 ts_tablet_manager.cc:1397] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b: Time spent bootstrapping tablet: real 0.085s user 0.056s sys 0.019s
I20250627 13:54:20.003098 12369 raft_consensus.cc:357] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:20.003350 12369 raft_consensus.cc:738] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 956ff058e6d64579a1963aabdb6d466b, State: Initialized, Role: LEARNER
I20250627 13:54:20.003625 12369 consensus_queue.cc:260] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:20.005431 12369 ts_tablet_manager.cc:1428] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250627 13:54:20.007200 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session 956ff058e6d64579a1963aabdb6d466b-eed3236c4fe8455cb9527b9f4b754e81 received from {username='slave'} at 127.0.103.70:59981
I20250627 13:54:20.007331 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session 956ff058e6d64579a1963aabdb6d466b-eed3236c4fe8455cb9527b9f4b754e81 on tablet eed3236c4fe8455cb9527b9f4b754e81 with peer 956ff058e6d64579a1963aabdb6d466b
I20250627 13:54:20.007397 12238 tablet_copy_client.cc:806] T ff83fdbab2b9417eac50495c54f0da54 P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:20.007830 12238 tablet_copy_client.cc:670] T ff83fdbab2b9417eac50495c54f0da54 P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:20.010180 12238 tablet_copy_client.cc:538] T ff83fdbab2b9417eac50495c54f0da54 P dede81a8c0164427a61532d01b6fc157: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:20.013486 12238 tablet_bootstrap.cc:492] T ff83fdbab2b9417eac50495c54f0da54 P dede81a8c0164427a61532d01b6fc157: Bootstrap starting.
I20250627 13:54:20.016652 12461 tablet_copy_client.cc:806] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:20.016883 12461 tablet_copy_client.cc:670] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:20.021735 12461 tablet_copy_client.cc:538] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:20.027297 12461 tablet_bootstrap.cc:492] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe: Bootstrap starting.
I20250627 13:54:20.115912 12339 tablet_bootstrap.cc:492] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:20.116333 12339 tablet_bootstrap.cc:492] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap complete.
I20250627 13:54:20.116609 12339 ts_tablet_manager.cc:1397] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b: Time spent bootstrapping tablet: real 0.182s user 0.047s sys 0.008s
I20250627 13:54:20.117439 12339 raft_consensus.cc:357] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:20.117697 12339 raft_consensus.cc:738] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 956ff058e6d64579a1963aabdb6d466b, State: Initialized, Role: LEARNER
I20250627 13:54:20.117908 12339 consensus_queue.cc:260] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:20.118484 12339 ts_tablet_manager.cc:1428] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20250627 13:54:20.119180 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session 956ff058e6d64579a1963aabdb6d466b-5184c3f8e465472e9a98c14f8e703415 received from {username='slave'} at 127.0.103.70:59981
I20250627 13:54:20.119356 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session 956ff058e6d64579a1963aabdb6d466b-5184c3f8e465472e9a98c14f8e703415 on tablet 5184c3f8e465472e9a98c14f8e703415 with peer 956ff058e6d64579a1963aabdb6d466b
I20250627 13:54:20.133984 12427 tablet_bootstrap.cc:492] T 9196497cb11f4ccbbd62695253d4c8c9 P 6686c6c365bf4f4582058890161676fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:20.134471 12427 tablet_bootstrap.cc:492] T 9196497cb11f4ccbbd62695253d4c8c9 P 6686c6c365bf4f4582058890161676fe: Bootstrap complete.
I20250627 13:54:20.134781 12427 ts_tablet_manager.cc:1397] T 9196497cb11f4ccbbd62695253d4c8c9 P 6686c6c365bf4f4582058890161676fe: Time spent bootstrapping tablet: real 0.275s user 0.056s sys 0.014s
I20250627 13:54:20.135532 12427 raft_consensus.cc:357] T 9196497cb11f4ccbbd62695253d4c8c9 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:20.135763 12427 raft_consensus.cc:738] T 9196497cb11f4ccbbd62695253d4c8c9 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6686c6c365bf4f4582058890161676fe, State: Initialized, Role: LEARNER
I20250627 13:54:20.135962 12427 consensus_queue.cc:260] T 9196497cb11f4ccbbd62695253d4c8c9 P 6686c6c365bf4f4582058890161676fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:20.137694 12427 ts_tablet_manager.cc:1428] T 9196497cb11f4ccbbd62695253d4c8c9 P 6686c6c365bf4f4582058890161676fe: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250627 13:54:20.138586 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session 6686c6c365bf4f4582058890161676fe-9196497cb11f4ccbbd62695253d4c8c9 received from {username='slave'} at 127.0.103.71:59981
I20250627 13:54:20.138726 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session 6686c6c365bf4f4582058890161676fe-9196497cb11f4ccbbd62695253d4c8c9 on tablet 9196497cb11f4ccbbd62695253d4c8c9 with peer 6686c6c365bf4f4582058890161676fe
I20250627 13:54:20.161984 10104 raft_consensus.cc:1215] T 03640a5124d2405faf4a26d0e22e1ffc P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20250627 13:54:20.172516 9711 raft_consensus.cc:1215] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250627 13:54:20.230711 12339 ts_tablet_manager.cc:927] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b: Initiating tablet copy from peer 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:54:20.237128 9973 raft_consensus.cc:1215] T 9196497cb11f4ccbbd62695253d4c8c9 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250627 13:54:20.246884 12339 tablet_copy_client.cc:323] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:37467
I20250627 13:54:20.253070 12238 tablet_bootstrap.cc:492] T ff83fdbab2b9417eac50495c54f0da54 P dede81a8c0164427a61532d01b6fc157: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:20.253607 12238 tablet_bootstrap.cc:492] T ff83fdbab2b9417eac50495c54f0da54 P dede81a8c0164427a61532d01b6fc157: Bootstrap complete.
I20250627 13:54:20.258229 12238 ts_tablet_manager.cc:1397] T ff83fdbab2b9417eac50495c54f0da54 P dede81a8c0164427a61532d01b6fc157: Time spent bootstrapping tablet: real 0.245s user 0.060s sys 0.009s
I20250627 13:54:20.259217 12238 raft_consensus.cc:357] T ff83fdbab2b9417eac50495c54f0da54 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:20.259459 12238 raft_consensus.cc:738] T ff83fdbab2b9417eac50495c54f0da54 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dede81a8c0164427a61532d01b6fc157, State: Initialized, Role: LEARNER
I20250627 13:54:20.259686 12238 consensus_queue.cc:260] T ff83fdbab2b9417eac50495c54f0da54 P dede81a8c0164427a61532d01b6fc157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:20.262768 12238 ts_tablet_manager.cc:1428] T ff83fdbab2b9417eac50495c54f0da54 P dede81a8c0164427a61532d01b6fc157: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250627 13:54:20.263666 9840 raft_consensus.cc:1215] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250627 13:54:20.264439 7822 tablet_copy_service.cc:342] P 83574c1bf8ea45adaff939158f93c096: Request end of tablet copy session dede81a8c0164427a61532d01b6fc157-ff83fdbab2b9417eac50495c54f0da54 received from {username='slave'} at 127.0.103.72:38213
I20250627 13:54:20.264578 7822 tablet_copy_service.cc:434] P 83574c1bf8ea45adaff939158f93c096: ending tablet copy session dede81a8c0164427a61532d01b6fc157-ff83fdbab2b9417eac50495c54f0da54 on tablet ff83fdbab2b9417eac50495c54f0da54 with peer dede81a8c0164427a61532d01b6fc157
I20250627 13:54:20.276742 11862 raft_consensus.cc:1062] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: attempting to promote NON_VOTER 956ff058e6d64579a1963aabdb6d466b to VOTER
I20250627 13:54:20.277957 11862 consensus_queue.cc:237] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:20.288707 9840 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250627 13:54:20.289275 12268 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:20.289868 9973 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250627 13:54:20.290725 12268 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:20.296976 9711 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250627 13:54:20.297394 12238 ts_tablet_manager.cc:927] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:20.297943 12268 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:20.312903 12238 tablet_copy_client.cc:323] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:20.313812 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet 83cfc975906745a09c5e1dcfac812ae2 from peer dede81a8c0164427a61532d01b6fc157 ({username='slave'} at 127.0.103.72:50607)
I20250627 13:54:20.314028 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet 83cfc975906745a09c5e1dcfac812ae2 from peer dede81a8c0164427a61532d01b6fc157 at {username='slave'} at 127.0.103.72:50607: session id = dede81a8c0164427a61532d01b6fc157-83cfc975906745a09c5e1dcfac812ae2
I20250627 13:54:20.314653 12268 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:20.317046 8082 tablet_copy_source_session.cc:215] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:20.318449 12238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83cfc975906745a09c5e1dcfac812ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:20.318581 7822 tablet_copy_service.cc:140] P 83574c1bf8ea45adaff939158f93c096: Received BeginTabletCopySession request for tablet 1dc569a7b12d40058d073f7b81b2601c from peer 956ff058e6d64579a1963aabdb6d466b ({username='slave'} at 127.0.103.70:37961)
I20250627 13:54:20.318740 7822 tablet_copy_service.cc:161] P 83574c1bf8ea45adaff939158f93c096: Beginning new tablet copy session on tablet 1dc569a7b12d40058d073f7b81b2601c from peer 956ff058e6d64579a1963aabdb6d466b at {username='slave'} at 127.0.103.70:37961: session id = 956ff058e6d64579a1963aabdb6d466b-1dc569a7b12d40058d073f7b81b2601c
I20250627 13:54:20.322515 7822 tablet_copy_source_session.cc:215] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:20.323936 12339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dc569a7b12d40058d073f7b81b2601c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:20.324244 9840 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:20.331717 12238 tablet_copy_client.cc:806] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:20.332005 12238 tablet_copy_client.cc:670] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:20.333091 12339 tablet_copy_client.cc:806] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:20.333371 12339 tablet_copy_client.cc:670] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:20.334710 7653 catalog_manager.cc:5582] T e3053b529a264e8697a1df9e17eeea81 P 956ff058e6d64579a1963aabdb6d466b reported cstate change: config changed from index 11 to 12, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:20.335191 12238 tablet_copy_client.cc:538] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:20.343972 12339 tablet_copy_client.cc:538] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:20.344005 12238 tablet_bootstrap.cc:492] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157: Bootstrap starting.
I20250627 13:54:20.347273 9973 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:20.352557 12339 tablet_bootstrap.cc:492] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b: Bootstrap starting.
I20250627 13:54:20.354761 9711 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:20.414356 9840 raft_consensus.cc:1215] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20250627 13:54:20.425622 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:20.438760 12369 ts_tablet_manager.cc:927] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:20.449226 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:20.451211 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:20.478710 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:20.480890 10083 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:20.494556 12268 raft_consensus.cc:1062] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e: attempting to promote NON_VOTER 956ff058e6d64579a1963aabdb6d466b to VOTER
I20250627 13:54:20.495389 12268 consensus_queue.cc:237] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:20.520612 12461 tablet_bootstrap.cc:492] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:20.544565 9840 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:20.545496 11862 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:20.545871 12461 tablet_bootstrap.cc:492] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe: Bootstrap complete.
I20250627 13:54:20.546195 12461 ts_tablet_manager.cc:1397] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe: Time spent bootstrapping tablet: real 0.519s user 0.046s sys 0.014s
I20250627 13:54:20.547251 12461 raft_consensus.cc:357] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:20.547470 12461 raft_consensus.cc:738] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6686c6c365bf4f4582058890161676fe, State: Initialized, Role: LEARNER
I20250627 13:54:20.547727 12461 consensus_queue.cc:260] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:20.552063 12461 ts_tablet_manager.cc:1428] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250627 13:54:20.557582 10104 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:20.558434 12531 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:20.569023 7822 tablet_copy_service.cc:342] P 83574c1bf8ea45adaff939158f93c096: Request end of tablet copy session 6686c6c365bf4f4582058890161676fe-897206e017ff4305a8cc9092e30f46b6 received from {username='slave'} at 127.0.103.71:53529
I20250627 13:54:20.569128 7926 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:20.569173 7822 tablet_copy_service.cc:434] P 83574c1bf8ea45adaff939158f93c096: ending tablet copy session 6686c6c365bf4f4582058890161676fe-897206e017ff4305a8cc9092e30f46b6 on tablet 897206e017ff4305a8cc9092e30f46b6 with peer 6686c6c365bf4f4582058890161676fe
I20250627 13:54:20.569815 12531 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:20.571975 9973 raft_consensus.cc:1215] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250627 13:54:20.587671 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:20.588097 12115 raft_consensus.cc:1062] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854: attempting to promote NON_VOTER dede81a8c0164427a61532d01b6fc157 to VOTER
I20250627 13:54:20.589035 12115 consensus_queue.cc:237] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } }
I20250627 13:54:20.603914 10102 raft_consensus.cc:1215] T ff83fdbab2b9417eac50495c54f0da54 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250627 13:54:20.622591 12531 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:20.624557 7926 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:20.631810 12369 tablet_copy_client.cc:323] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:20.631916 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:20.632513 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet 83cfc975906745a09c5e1dcfac812ae2 from peer 956ff058e6d64579a1963aabdb6d466b ({username='slave'} at 127.0.103.70:59981)
I20250627 13:54:20.632720 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet 83cfc975906745a09c5e1dcfac812ae2 from peer 956ff058e6d64579a1963aabdb6d466b at {username='slave'} at 127.0.103.70:59981: session id = 956ff058e6d64579a1963aabdb6d466b-83cfc975906745a09c5e1dcfac812ae2
I20250627 13:54:20.633246 9841 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:20.633147 7658 catalog_manager.cc:5582] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 9 to 10, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:20.633829 12418 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:20.630713 9840 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:20.635326 10104 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:20.635896 12115 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:20.636762 8190 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:20.635999 8082 tablet_copy_source_session.cc:215] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:20.638231 12115 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:20.638712 10102 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:20.642798 12369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83cfc975906745a09c5e1dcfac812ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:20.649914 8189 consensus_queue.cc:237] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:20.654289 10102 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:20.654810 11862 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:20.665629 12531 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:20.670749 7648 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 14dba72d45694defb83f41c1572722ca with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:20.671500 10103 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:20.672938 7653 catalog_manager.cc:5582] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 10 to 11, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:20.679487 7911 tablet_service.cc:1515] Processing DeleteTablet for tablet 14dba72d45694defb83f41c1572722ca with delete_type TABLET_DATA_TOMBSTONED (TS a41a89ae8ed743edb2b5cf4490d8c854 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:59302
I20250627 13:54:20.680022 9839 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:20.680989 11862 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:20.685385 12369 tablet_copy_client.cc:806] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:20.697000 12418 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:20.700119 9840 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:20.704334 8189 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:20.705060 7653 catalog_manager.cc:5582] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index 10 to 11, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:20.706449 9842 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:20.711417 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:20.711512 12180 raft_consensus.cc:1062] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a to VOTER
I20250627 13:54:20.711829 10103 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:20.712350 12180 consensus_queue.cc:237] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:20.727599 12340 raft_consensus.cc:1062] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER 6686c6c365bf4f4582058890161676fe to VOTER
I20250627 13:54:20.728411 12340 consensus_queue.cc:237] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:20.733423 12369 tablet_copy_client.cc:670] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:20.734591 12585 tablet_replica.cc:331] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854: stopping tablet replica
I20250627 13:54:20.738057 12369 tablet_copy_client.cc:538] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:20.749692 9711 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:20.750396 11961 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:20.754500 12369 tablet_bootstrap.cc:492] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap starting.
I20250627 13:54:20.754726 9973 raft_consensus.cc:1273] T 9196497cb11f4ccbbd62695253d4c8c9 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:20.755282 11961 consensus_queue.cc:1035] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:20.761459 12585 raft_consensus.cc:2241] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:20.761874 12585 raft_consensus.cc:2270] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:20.762909 12585 ts_tablet_manager.cc:1905] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:20.770315 8190 raft_consensus.cc:1273] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:20.771014 12340 consensus_queue.cc:1035] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:20.773856 7800 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:20.774725 9839 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:20.775367 11961 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:20.776734 11961 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:20.789004 9842 raft_consensus.cc:1273] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:20.804658 12340 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:20.812309 11961 raft_consensus.cc:1062] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER 956ff058e6d64579a1963aabdb6d466b to VOTER
I20250627 13:54:20.813164 11961 consensus_queue.cc:237] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:20.815754 8190 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:20.817097 12180 consensus_queue.cc:1035] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:20.818207 9974 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:20.821013 12180 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:20.823868 7800 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:20.824626 7658 catalog_manager.cc:5582] T 9196497cb11f4ccbbd62695253d4c8c9 P 6686c6c365bf4f4582058890161676fe reported cstate change: config changed from index 7 to 8, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:20.830145 7657 catalog_manager.cc:5582] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 7 to 8, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:20.831919 9840 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:20.840965 9974 raft_consensus.cc:1273] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:20.842053 9839 raft_consensus.cc:1273] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:20.843058 12598 consensus_queue.cc:1035] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:20.846200 11961 consensus_queue.cc:1035] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:20.846539 7800 raft_consensus.cc:1273] T 5184c3f8e465472e9a98c14f8e703415 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:20.853768 12340 consensus_queue.cc:1035] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:20.853113 9842 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:20.880010 12598 raft_consensus.cc:1062] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER 956ff058e6d64579a1963aabdb6d466b to VOTER
I20250627 13:54:20.880901 12598 consensus_queue.cc:237] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:20.884925 12340 raft_consensus.cc:2953] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:20.887583 9973 raft_consensus.cc:2953] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:20.896679 12585 ts_tablet_manager.cc:1918] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20250627 13:54:20.897763 7800 raft_consensus.cc:2953] T 5184c3f8e465472e9a98c14f8e703415 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:20.938598 7660 catalog_manager.cc:5582] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe reported cstate change: config changed from index 7 to 8, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
W20250627 13:54:20.941013 7999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.71:37859, user_credentials={real_user=slave}} blocked reactor thread for 48046.5us
I20250627 13:54:20.965443 9711 raft_consensus.cc:1273] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:20.969744 12339 tablet_bootstrap.cc:492] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:20.970199 12339 tablet_bootstrap.cc:492] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b: Bootstrap complete.
I20250627 13:54:20.970475 12339 ts_tablet_manager.cc:1397] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b: Time spent bootstrapping tablet: real 0.618s user 0.057s sys 0.006s
I20250627 13:54:20.971697 9711 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:20.974349 12339 raft_consensus.cc:357] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:20.974575 12339 raft_consensus.cc:738] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 956ff058e6d64579a1963aabdb6d466b, State: Initialized, Role: LEARNER
I20250627 13:54:20.971025 12632 consensus_queue.cc:1035] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:20.974776 12339 consensus_queue.cc:260] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:20.980475 12238 tablet_bootstrap.cc:492] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:20.986400 12238 tablet_bootstrap.cc:492] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157: Bootstrap complete.
I20250627 13:54:20.988402 12339 ts_tablet_manager.cc:1428] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20250627 13:54:20.989197 12238 ts_tablet_manager.cc:1397] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157: Time spent bootstrapping tablet: real 0.646s user 0.066s sys 0.009s
I20250627 13:54:20.990895 12238 raft_consensus.cc:357] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:20.993252 7822 tablet_copy_service.cc:342] P 83574c1bf8ea45adaff939158f93c096: Request end of tablet copy session 956ff058e6d64579a1963aabdb6d466b-1dc569a7b12d40058d073f7b81b2601c received from {username='slave'} at 127.0.103.70:37961
I20250627 13:54:20.993377 7822 tablet_copy_service.cc:434] P 83574c1bf8ea45adaff939158f93c096: ending tablet copy session 956ff058e6d64579a1963aabdb6d466b-1dc569a7b12d40058d073f7b81b2601c on tablet 1dc569a7b12d40058d073f7b81b2601c with peer 956ff058e6d64579a1963aabdb6d466b
W20250627 13:54:20.994331 7998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.72:35213, user_credentials={real_user=slave}} blocked reactor thread for 103060us
I20250627 13:54:20.995626 12238 raft_consensus.cc:738] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dede81a8c0164427a61532d01b6fc157, State: Initialized, Role: LEARNER
I20250627 13:54:20.996091 12238 consensus_queue.cc:260] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:20.996670 12238 ts_tablet_manager.cc:1428] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250627 13:54:21.004457 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session dede81a8c0164427a61532d01b6fc157-83cfc975906745a09c5e1dcfac812ae2 received from {username='slave'} at 127.0.103.72:50607
I20250627 13:54:21.004635 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session dede81a8c0164427a61532d01b6fc157-83cfc975906745a09c5e1dcfac812ae2 on tablet 83cfc975906745a09c5e1dcfac812ae2 with peer dede81a8c0164427a61532d01b6fc157
I20250627 13:54:21.006923 12585 log.cc:1199] T 14dba72d45694defb83f41c1572722ca P a41a89ae8ed743edb2b5cf4490d8c854: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/wals/14dba72d45694defb83f41c1572722ca
I20250627 13:54:21.007854 7647 catalog_manager.cc:4928] TS a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675): tablet 14dba72d45694defb83f41c1572722ca (table rebalance_test_table_6 [id=f83cab74056a4e7681a701169d1c1716]) successfully deleted
I20250627 13:54:21.009305 7800 raft_consensus.cc:1273] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:21.011065 9842 raft_consensus.cc:1273] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:21.011842 9842 raft_consensus.cc:2953] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:21.023952 12638 consensus_queue.cc:1035] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:21.030467 12340 consensus_queue.cc:1035] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:21.064189 12340 raft_consensus.cc:2953] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:21.070088 8062 consensus_queue.cc:237] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:21.073536 7800 raft_consensus.cc:2953] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:21.076608 9711 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:21.077342 12180 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:21.083297 9709 raft_consensus.cc:2953] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:21.093264 9839 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:21.094302 12640 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:21.099224 12180 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:21.103492 9708 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:21.107782 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:21.112710 7653 catalog_manager.cc:5582] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: config changed from index 8 to 9, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:21.114105 7660 catalog_manager.cc:5582] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 8 to 9, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:21.122223 7782 tablet_service.cc:1515] Processing DeleteTablet for tablet e2bd060a4a9e4765b45b0242ae56e489 with delete_type TABLET_DATA_TOMBSTONED (TS 83574c1bf8ea45adaff939158f93c096 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:45826
I20250627 13:54:21.133338 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet e2bd060a4a9e4765b45b0242ae56e489 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:21.140599 9842 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:21.147037 9842 raft_consensus.cc:2953] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:21.153991 12564 raft_consensus.cc:1062] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096: attempting to promote NON_VOTER 6686c6c365bf4f4582058890161676fe to VOTER
I20250627 13:54:21.154969 12564 consensus_queue.cc:237] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:21.170787 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:21.190542 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:21.204563 12570 raft_consensus.cc:1062] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096: attempting to promote NON_VOTER dede81a8c0164427a61532d01b6fc157 to VOTER
I20250627 13:54:21.205514 12570 consensus_queue.cc:237] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } }
I20250627 13:54:21.232971 12666 tablet_replica.cc:331] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096: stopping tablet replica
I20250627 13:54:21.233331 12666 raft_consensus.cc:2241] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:21.235447 9973 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe [term 1 LEARNER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:21.236285 12563 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:21.243124 10103 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:21.245062 12563 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:21.272773 10104 raft_consensus.cc:1215] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20250627 13:54:21.281154 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:21.283725 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:21.290551 12666 raft_consensus.cc:2270] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:21.294826 9842 raft_consensus.cc:1215] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250627 13:54:21.311372 12666 ts_tablet_manager.cc:1905] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:21.313136 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:21.320617 9974 raft_consensus.cc:1273] T ff83fdbab2b9417eac50495c54f0da54 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:21.322646 12563 consensus_queue.cc:1035] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250627 13:54:21.326339 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:21.326584 8190 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:21.326730 10104 raft_consensus.cc:1273] T ff83fdbab2b9417eac50495c54f0da54 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:21.327077 12564 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:21.331844 7926 raft_consensus.cc:1273] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:21.338420 12563 consensus_queue.cc:1035] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:21.339035 12563 consensus_queue.cc:1035] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:21.375178 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:21.396142 8062 consensus_queue.cc:237] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:21.398788 9842 raft_consensus.cc:1273] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:21.399513 12634 consensus_queue.cc:1035] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:21.402299 9972 raft_consensus.cc:1273] T 9196497cb11f4ccbbd62695253d4c8c9 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:21.402925 12180 consensus_queue.cc:1035] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:21.405818 12180 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:21.412357 9842 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:21.416462 9972 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:21.429371 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9196497cb11f4ccbbd62695253d4c8c9 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:21.432138 12369 tablet_bootstrap.cc:492] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:21.432055 7660 catalog_manager.cc:5582] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 8 to 9, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:21.432550 12369 tablet_bootstrap.cc:492] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap complete.
I20250627 13:54:21.432781 12369 ts_tablet_manager.cc:1397] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b: Time spent bootstrapping tablet: real 0.678s user 0.052s sys 0.003s
I20250627 13:54:21.439985 8171 tablet_service.cc:1515] Processing DeleteTablet for tablet 9196497cb11f4ccbbd62695253d4c8c9 with delete_type TABLET_DATA_TOMBSTONED (TS e4c831df4da841929d13d6c783099b0e not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:48616
I20250627 13:54:21.454871 12369 raft_consensus.cc:357] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:21.455085 12369 raft_consensus.cc:738] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 956ff058e6d64579a1963aabdb6d466b, State: Initialized, Role: LEARNER
I20250627 13:54:21.455226 12369 consensus_queue.cc:260] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:21.455737 12369 ts_tablet_manager.cc:1428] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b: Time spent starting tablet: real 0.023s user 0.003s sys 0.000s
I20250627 13:54:21.456934 12714 tablet_replica.cc:331] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e: stopping tablet replica
I20250627 13:54:21.460005 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session 956ff058e6d64579a1963aabdb6d466b-83cfc975906745a09c5e1dcfac812ae2 received from {username='slave'} at 127.0.103.70:59981
I20250627 13:54:21.460155 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session 956ff058e6d64579a1963aabdb6d466b-83cfc975906745a09c5e1dcfac812ae2 on tablet 83cfc975906745a09c5e1dcfac812ae2 with peer 956ff058e6d64579a1963aabdb6d466b
I20250627 13:54:21.461447 12714 raft_consensus.cc:2241] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:21.462788 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:21.463627 12714 raft_consensus.cc:2270] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:21.465335 12714 ts_tablet_manager.cc:1905] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:21.479177 9842 raft_consensus.cc:1215] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20250627 13:54:21.500399 12570 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:21.503921 9972 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:21.506063 12714 ts_tablet_manager.cc:1918] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20250627 13:54:21.506896 8190 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:21.508659 12714 log.cc:1199] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal/wals/9196497cb11f4ccbbd62695253d4c8c9
I20250627 13:54:21.511512 7648 catalog_manager.cc:4928] TS e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): tablet 9196497cb11f4ccbbd62695253d4c8c9 (table rebalance_test_table_4 [id=4d92d465970b4ec4b5eace263b25362a]) successfully deleted
I20250627 13:54:21.509130 7653 catalog_manager.cc:5582] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 7 to 8, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:21.525708 10103 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:21.528754 7800 consensus_queue.cc:237] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:21.557838 12669 raft_consensus.cc:2953] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:21.561652 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:21.564030 7927 raft_consensus.cc:2953] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:21.566306 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:21.571327 10102 raft_consensus.cc:2953] T ff83fdbab2b9417eac50495c54f0da54 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:21.589598 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:21.594152 7660 catalog_manager.cc:5582] T ff83fdbab2b9417eac50495c54f0da54 P dede81a8c0164427a61532d01b6fc157 reported cstate change: config changed from index 7 to 8, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:21.616374 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:21.623462 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
W20250627 13:54:21.640415 12702 log.cc:927] Time spent T ff83fdbab2b9417eac50495c54f0da54 P 6686c6c365bf4f4582058890161676fe: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20250627 13:54:21.664366 9971 raft_consensus.cc:2953] T ff83fdbab2b9417eac50495c54f0da54 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:21.691272 9973 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:21.692162 12570 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20250627 13:54:21.699437 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:21.706974 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:21.709983 12563 raft_consensus.cc:1062] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096: attempting to promote NON_VOTER 956ff058e6d64579a1963aabdb6d466b to VOTER
I20250627 13:54:21.710875 12563 consensus_queue.cc:237] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:21.721796 7927 raft_consensus.cc:1273] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:21.722447 12564 consensus_queue.cc:1035] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:21.725826 9842 raft_consensus.cc:1273] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:21.726351 12564 consensus_queue.cc:1035] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:21.731053 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:21.733100 9709 raft_consensus.cc:1273] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:21.733676 12570 consensus_queue.cc:1035] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:21.739341 7800 consensus_queue.cc:237] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } }
I20250627 13:54:21.745869 10102 raft_consensus.cc:1273] T ff83fdbab2b9417eac50495c54f0da54 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:21.746461 12564 consensus_queue.cc:1035] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:21.748610 12564 raft_consensus.cc:2953] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:21.753507 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet ff83fdbab2b9417eac50495c54f0da54 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:21.754392 9974 raft_consensus.cc:1273] T ff83fdbab2b9417eac50495c54f0da54 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:21.754331 10102 raft_consensus.cc:2953] T ff83fdbab2b9417eac50495c54f0da54 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:21.763418 7653 catalog_manager.cc:5582] T ff83fdbab2b9417eac50495c54f0da54 P dede81a8c0164427a61532d01b6fc157 reported cstate change: config changed from index 8 to 9, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:21.768026 7911 tablet_service.cc:1515] Processing DeleteTablet for tablet ff83fdbab2b9417eac50495c54f0da54 with delete_type TABLET_DATA_TOMBSTONED (TS a41a89ae8ed743edb2b5cf4490d8c854 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:59302
I20250627 13:54:21.777151 12669 consensus_queue.cc:1035] T ff83fdbab2b9417eac50495c54f0da54 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:21.779361 8041 tablet_service.cc:1430] Tablet server has 13 leaders and 0 scanners
I20250627 13:54:21.786067 12720 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:21.789638 10083 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:21.807144 7660 catalog_manager.cc:5582] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 8 to 9, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:21.809984 10104 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:21.825915 12180 raft_consensus.cc:1062] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER dede81a8c0164427a61532d01b6fc157 to VOTER
I20250627 13:54:21.826819 12180 consensus_queue.cc:237] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
W20250627 13:54:21.836865 8126 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.67:38933, user_credentials={real_user=slave}} blocked reactor thread for 68308us
W20250627 13:54:21.838829 7737 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.72:35213, user_credentials={real_user=slave}} blocked reactor thread for 77001.1us
I20250627 13:54:21.839376 12563 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20250627 13:54:21.840945 7738 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.71:37859, user_credentials={real_user=slave}} blocked reactor thread for 82992.4us
I20250627 13:54:21.841851 12563 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:21.842654 9822 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:21.849663 9842 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:21.850554 9973 raft_consensus.cc:2953] T ff83fdbab2b9417eac50495c54f0da54 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:21.852464 7660 catalog_manager.cc:5582] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 7 to 8, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:21.854560 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 897206e017ff4305a8cc9092e30f46b6 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:21.856582 9972 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:21.866472 7800 consensus_queue.cc:237] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:21.866657 10103 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:21.871356 9840 raft_consensus.cc:1273] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:21.872032 12638 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:21.873631 12666 ts_tablet_manager.cc:1918] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20250627 13:54:21.873772 12666 log.cc:1199] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/wals/e2bd060a4a9e4765b45b0242ae56e489
I20250627 13:54:21.874791 7645 catalog_manager.cc:4928] TS 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467): tablet e2bd060a4a9e4765b45b0242ae56e489 (table rebalance_test_table_6 [id=f83cab74056a4e7681a701169d1c1716]) successfully deleted
I20250627 13:54:21.871356 9842 raft_consensus.cc:1273] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:21.878769 12733 consensus_queue.cc:1035] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:21.886987 12638 raft_consensus.cc:1025] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: attempt to promote peer 956ff058e6d64579a1963aabdb6d466b: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250627 13:54:21.904326 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:21.909124 7931 raft_consensus.cc:1273] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:21.909796 12733 consensus_queue.cc:1035] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:21.913044 7933 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:21.917160 7931 raft_consensus.cc:1273] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:21.917783 12180 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:21.922050 12733 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:21.926533 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 1dc569a7b12d40058d073f7b81b2601c with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:21.926406 7927 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:21.929113 7660 catalog_manager.cc:5582] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 8 to 9, VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) evicted. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:21.933175 9691 tablet_service.cc:1515] Processing DeleteTablet for tablet 1dc569a7b12d40058d073f7b81b2601c with delete_type TABLET_DATA_TOMBSTONED (TS 2a1282ebe92a4c9a92eed3b13f69012a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:53454
I20250627 13:54:21.936455 12734 tablet_replica.cc:331] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854: stopping tablet replica
I20250627 13:54:21.936723 12734 raft_consensus.cc:2241] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:21.936973 12734 raft_consensus.cc:2270] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:21.942813 9841 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:21.958238 12734 ts_tablet_manager.cc:1905] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:21.958863 8171 tablet_service.cc:1515] Processing DeleteTablet for tablet 897206e017ff4305a8cc9092e30f46b6 with delete_type TABLET_DATA_TOMBSTONED (TS e4c831df4da841929d13d6c783099b0e not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:48616
I20250627 13:54:21.959311 12714 tablet_replica.cc:331] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e: stopping tablet replica
I20250627 13:54:21.959565 12714 raft_consensus.cc:2241] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:21.959820 12714 raft_consensus.cc:2270] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:21.960724 12714 ts_tablet_manager.cc:1905] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:21.968001 12714 ts_tablet_manager.cc:1918] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20250627 13:54:21.968134 12714 log.cc:1199] T 897206e017ff4305a8cc9092e30f46b6 P e4c831df4da841929d13d6c783099b0e: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal/wals/897206e017ff4305a8cc9092e30f46b6
I20250627 13:54:21.970386 8190 raft_consensus.cc:1273] T 83cfc975906745a09c5e1dcfac812ae2 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:21.970567 12747 tablet_replica.cc:331] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a: stopping tablet replica
I20250627 13:54:21.971000 12747 raft_consensus.cc:2241] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:21.971257 12747 pending_rounds.cc:70] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a: Trying to abort 1 pending ops.
I20250627 13:54:21.971657 12180 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:21.971531 12747 pending_rounds.cc:77] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7172228963165093888 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "1dc569a7b12d40058d073f7b81b2601c" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } } }
I20250627 13:54:21.972301 8190 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5f176b66fdce4398be3492a5f3b4ea27"
dest_uuid: "e4c831df4da841929d13d6c783099b0e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59012
I20250627 13:54:21.972527 8190 raft_consensus.cc:604] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Received request to transfer leadership
I20250627 13:54:21.973131 12747 raft_consensus.cc:2915] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20250627 13:54:21.973534 12747 raft_consensus.cc:2887] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250627 13:54:21.973794 12747 raft_consensus.cc:2270] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:21.983168 10104 raft_consensus.cc:1273] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157 [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250627 13:54:21.984172 7648 catalog_manager.cc:4928] TS e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): tablet 897206e017ff4305a8cc9092e30f46b6 (table rebalance_test_table_4 [id=4d92d465970b4ec4b5eace263b25362a]) successfully deleted
I20250627 13:54:21.984761 12634 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:21.986904 12735 raft_consensus.cc:991] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e: : Instructing follower dede81a8c0164427a61532d01b6fc157 to start an election
I20250627 13:54:21.987094 12736 raft_consensus.cc:1079] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Signalling peer dede81a8c0164427a61532d01b6fc157 to start an election
I20250627 13:54:21.991303 10102 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "5f176b66fdce4398be3492a5f3b4ea27"
dest_uuid: "dede81a8c0164427a61532d01b6fc157"
from {username='slave'} at 127.0.103.68:39671
I20250627 13:54:21.991607 10102 raft_consensus.cc:491] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:54:21.991847 10102 raft_consensus.cc:3058] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:21.995214 12734 ts_tablet_manager.cc:1918] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20250627 13:54:21.995347 12734 log.cc:1199] T ff83fdbab2b9417eac50495c54f0da54 P a41a89ae8ed743edb2b5cf4490d8c854: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/wals/ff83fdbab2b9417eac50495c54f0da54
I20250627 13:54:21.996155 7647 catalog_manager.cc:4928] TS a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675): tablet ff83fdbab2b9417eac50495c54f0da54 (table rebalance_test_table_8 [id=b84f3fe3fd7c4f01902547d2db982f90]) successfully deleted
I20250627 13:54:21.996136 10102 raft_consensus.cc:513] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } }
I20250627 13:54:21.999850 10102 leader_election.cc:290] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [CANDIDATE]: Term 2 election: Requested vote from peers 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467), e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:22.003260 8062 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f176b66fdce4398be3492a5f3b4ea27" candidate_uuid: "dede81a8c0164427a61532d01b6fc157" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660"
I20250627 13:54:22.003520 8062 raft_consensus.cc:3058] T 5f176b66fdce4398be3492a5f3b4ea27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:22.006613 10083 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:22.007555 8062 raft_consensus.cc:2466] T 5f176b66fdce4398be3492a5f3b4ea27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dede81a8c0164427a61532d01b6fc157 in term 2.
I20250627 13:54:22.011351 7800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f176b66fdce4398be3492a5f3b4ea27" candidate_uuid: "dede81a8c0164427a61532d01b6fc157" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "83574c1bf8ea45adaff939158f93c096"
I20250627 13:54:22.011583 7800 raft_consensus.cc:3058] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:22.014835 7800 raft_consensus.cc:2466] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dede81a8c0164427a61532d01b6fc157 in term 2.
I20250627 13:54:22.015277 10037 leader_election.cc:304] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, 9cf479bdcd0242738d2a77db4ad9f660, dede81a8c0164427a61532d01b6fc157; no voters:
I20250627 13:54:22.015646 12292 raft_consensus.cc:2802] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:54:22.040096 12747 ts_tablet_manager.cc:1905] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:22.053503 8189 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f176b66fdce4398be3492a5f3b4ea27" candidate_uuid: "dede81a8c0164427a61532d01b6fc157" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e4c831df4da841929d13d6c783099b0e"
I20250627 13:54:22.053776 8189 raft_consensus.cc:3053] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Stepping down as leader of term 1
I20250627 13:54:22.053879 8189 raft_consensus.cc:738] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Running, Role: LEADER
I20250627 13:54:22.054183 8189 consensus_queue.cc:260] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } }
I20250627 13:54:22.054589 8189 raft_consensus.cc:3058] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:22.057958 8189 raft_consensus.cc:2466] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dede81a8c0164427a61532d01b6fc157 in term 2.
I20250627 13:54:22.071799 12638 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:22.076467 9841 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:22.078845 7660 catalog_manager.cc:5582] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 5 to 6, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20250627 13:54:22.082211 12638 raft_consensus.cc:1062] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER 956ff058e6d64579a1963aabdb6d466b to VOTER
I20250627 13:54:22.083243 12638 consensus_queue.cc:237] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:22.085599 9840 raft_consensus.cc:1273] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:22.086189 12638 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:22.086521 8190 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:22.091877 7931 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:22.103386 7927 raft_consensus.cc:1273] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:22.104254 12757 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:22.109593 12757 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:22.110168 7927 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:22.115345 9840 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:22.121693 7657 catalog_manager.cc:5582] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b reported cstate change: config changed from index 6 to 7, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:22.139304 10104 raft_consensus.cc:1273] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:22.139868 12757 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:22.141057 10104 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:22.151872 12292 raft_consensus.cc:695] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [term 2 LEADER]: Becoming Leader. State: Replica: dede81a8c0164427a61532d01b6fc157, State: Running, Role: LEADER
I20250627 13:54:22.152379 12292 consensus_queue.cc:237] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } }
I20250627 13:54:22.154789 10104 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:22.171492 7660 catalog_manager.cc:5582] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 reported cstate change: term changed from 1 to 2, leader changed from e4c831df4da841929d13d6c783099b0e (127.0.103.68) to dede81a8c0164427a61532d01b6fc157 (127.0.103.72). New cstate: current_term: 2 leader_uuid: "dede81a8c0164427a61532d01b6fc157" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:22.176175 10083 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:22.199676 8189 raft_consensus.cc:1273] T 83cfc975906745a09c5e1dcfac812ae2 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:22.200623 12757 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:22.203351 8189 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:22.210731 8171 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:22.260896 12747 ts_tablet_manager.cc:1918] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20250627 13:54:22.261143 12747 log.cc:1199] T 1dc569a7b12d40058d073f7b81b2601c P 2a1282ebe92a4c9a92eed3b13f69012a: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-4/wal/wals/1dc569a7b12d40058d073f7b81b2601c
I20250627 13:54:22.262140 7648 catalog_manager.cc:4928] TS 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101): tablet 1dc569a7b12d40058d073f7b81b2601c (table rebalance_test_table_3 [id=ec6a9ed6ab4d416988b2abe2f8d29c55]) successfully deleted
I20250627 13:54:22.558866 8189 consensus_queue.cc:237] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } }
I20250627 13:54:22.573239 8062 consensus_queue.cc:237] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:22.581900 9840 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250627 13:54:22.583429 9972 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250627 13:54:22.583803 12815 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250627 13:54:22.584676 12735 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20250627 13:54:22.586797 7996 consensus_peers.cc:487] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:22.590649 9974 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:22.592586 8190 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:22.594309 12638 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:22.594995 12638 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:22.609809 7800 raft_consensus.cc:1273] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Refusing update from remote peer dede81a8c0164427a61532d01b6fc157: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20250627 13:54:22.612912 12292 consensus_queue.cc:1035] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20250627 13:54:22.615211 9711 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250627 13:54:22.615923 12735 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250627 13:54:22.644402 12735 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } }
I20250627 13:54:22.650107 9972 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } }
I20250627 13:54:22.650689 12180 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:22.656317 9973 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:22.669483 7653 catalog_manager.cc:5582] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 8 to 9, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:22.675653 7660 catalog_manager.cc:5582] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 12 to 13. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:22.686635 8062 raft_consensus.cc:1273] T 5f176b66fdce4398be3492a5f3b4ea27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Refusing update from remote peer dede81a8c0164427a61532d01b6fc157: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20250627 13:54:22.687826 12292 consensus_queue.cc:1035] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:22.692605 8190 consensus_queue.cc:237] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } }
W20250627 13:54:22.696348 8128 consensus_peers.cc:487] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e -> Peer a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675): Couldn't send request to peer a41a89ae8ed743edb2b5cf4490d8c854. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e3053b529a264e8697a1df9e17eeea81. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:22.698243 9711 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } }
I20250627 13:54:22.699949 9972 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250627 13:54:22.700489 8063 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "c2468f4746224f0ab6f4ba4b3a628e35"
dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52540
I20250627 13:54:22.700778 8063 raft_consensus.cc:604] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Received request to transfer leadership
I20250627 13:54:22.702602 9840 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250627 13:54:22.704401 12735 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250627 13:54:22.706195 12735 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20250627 13:54:22.711532 8190 raft_consensus.cc:1273] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Refusing update from remote peer dede81a8c0164427a61532d01b6fc157: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20250627 13:54:22.713390 12292 consensus_queue.cc:1035] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:22.715216 8189 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:22.717331 9840 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } }
I20250627 13:54:22.720559 12180 raft_consensus.cc:991] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660: : Instructing follower 83574c1bf8ea45adaff939158f93c096 to start an election
I20250627 13:54:22.720721 12180 raft_consensus.cc:1079] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Signalling peer 83574c1bf8ea45adaff939158f93c096 to start an election
I20250627 13:54:22.721930 7800 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "c2468f4746224f0ab6f4ba4b3a628e35"
dest_uuid: "83574c1bf8ea45adaff939158f93c096"
from {username='slave'} at 127.0.103.67:60539
I20250627 13:54:22.722208 7800 raft_consensus.cc:491] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:54:22.722376 7800 raft_consensus.cc:3058] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:22.726945 7800 raft_consensus.cc:513] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:22.728947 9972 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2468f4746224f0ab6f4ba4b3a628e35" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "6686c6c365bf4f4582058890161676fe"
I20250627 13:54:22.731037 8062 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2468f4746224f0ab6f4ba4b3a628e35" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660"
I20250627 13:54:22.731316 8062 raft_consensus.cc:3053] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Stepping down as leader of term 1
I20250627 13:54:22.731518 8062 raft_consensus.cc:738] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Running, Role: LEADER
I20250627 13:54:22.731999 8062 consensus_queue.cc:260] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:22.730036 9972 raft_consensus.cc:3058] T c2468f4746224f0ab6f4ba4b3a628e35 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:22.736001 9972 raft_consensus.cc:2466] T c2468f4746224f0ab6f4ba4b3a628e35 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 2.
I20250627 13:54:22.736920 8189 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2468f4746224f0ab6f4ba4b3a628e35" candidate_uuid: "83574c1bf8ea45adaff939158f93c096" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e4c831df4da841929d13d6c783099b0e"
I20250627 13:54:22.737181 8189 raft_consensus.cc:3058] T c2468f4746224f0ab6f4ba4b3a628e35 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:22.739579 7800 leader_election.cc:290] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 2 election: Requested vote from peers e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933), 6686c6c365bf4f4582058890161676fe (127.0.103.71:37859)
I20250627 13:54:22.750599 8190 consensus_queue.cc:237] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } }
I20250627 13:54:22.745147 8063 consensus_queue.cc:237] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:22.755656 7927 raft_consensus.cc:1273] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:22.756260 12638 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:22.757611 10102 raft_consensus.cc:1273] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:22.760185 12638 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20250627 13:54:22.761982 8128 consensus_peers.cc:487] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e -> Peer a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675): Couldn't send request to peer a41a89ae8ed743edb2b5cf4490d8c854. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:22.765775 9841 raft_consensus.cc:1273] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:22.767069 9972 raft_consensus.cc:1273] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:22.767671 12735 consensus_queue.cc:1035] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:22.770080 12757 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:22.774068 9841 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:22.783293 8062 raft_consensus.cc:3058] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:22.802250 8189 raft_consensus.cc:2466] T c2468f4746224f0ab6f4ba4b3a628e35 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 2.
I20250627 13:54:22.802870 7738 leader_election.cc:304] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6686c6c365bf4f4582058890161676fe, 83574c1bf8ea45adaff939158f93c096, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:54:22.803383 12570 raft_consensus.cc:2802] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:54:22.803961 12570 raft_consensus.cc:695] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096 [term 2 LEADER]: Becoming Leader. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Running, Role: LEADER
I20250627 13:54:22.804314 12570 consensus_queue.cc:237] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:22.815577 7660 catalog_manager.cc:5582] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b reported cstate change: config changed from index 7 to 8, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:22.819741 8171 tablet_service.cc:1515] Processing DeleteTablet for tablet 83cfc975906745a09c5e1dcfac812ae2 with delete_type TABLET_DATA_TOMBSTONED (TS e4c831df4da841929d13d6c783099b0e not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:48616
I20250627 13:54:22.819936 8062 raft_consensus.cc:2466] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83574c1bf8ea45adaff939158f93c096 in term 2.
I20250627 13:54:22.823967 12735 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:22.829244 7648 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e3053b529a264e8697a1df9e17eeea81 with cas_config_opid_index 13: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250627 13:54:22.833146 9840 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:22.835582 12858 tablet_replica.cc:331] T 83cfc975906745a09c5e1dcfac812ae2 P e4c831df4da841929d13d6c783099b0e: stopping tablet replica
I20250627 13:54:22.835886 12858 raft_consensus.cc:2241] T 83cfc975906745a09c5e1dcfac812ae2 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:22.836287 12858 raft_consensus.cc:2270] T 83cfc975906745a09c5e1dcfac812ae2 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:22.837489 12858 ts_tablet_manager.cc:1905] T 83cfc975906745a09c5e1dcfac812ae2 P e4c831df4da841929d13d6c783099b0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:22.845582 9974 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:22.846575 7660 catalog_manager.cc:5582] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 13 to 14, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:22.861253 12858 ts_tablet_manager.cc:1918] T 83cfc975906745a09c5e1dcfac812ae2 P e4c831df4da841929d13d6c783099b0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
W20250627 13:54:22.863339 10037 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.67:38933, user_credentials={real_user=slave}} blocked reactor thread for 136259us
W20250627 13:54:22.864537 7998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.66:43675, user_credentials={real_user=slave}} blocked reactor thread for 101377us
I20250627 13:54:22.864537 7660 catalog_manager.cc:5582] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096 reported cstate change: term changed from 1 to 2, leader changed from 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) to 83574c1bf8ea45adaff939158f93c096 (127.0.103.65). New cstate: current_term: 2 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:22.864642 12858 log.cc:1199] T 83cfc975906745a09c5e1dcfac812ae2 P e4c831df4da841929d13d6c783099b0e: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal/wals/83cfc975906745a09c5e1dcfac812ae2
I20250627 13:54:22.866034 7927 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:22.867583 7648 catalog_manager.cc:4928] TS e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): tablet 83cfc975906745a09c5e1dcfac812ae2 (table rebalance_test_table_9 [id=19e5686b50fe4187a733aa6413ef2ce0]) successfully deleted
I20250627 13:54:22.877312 9709 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250627 13:54:22.878073 12860 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250627 13:54:22.879958 9711 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:22.888540 8062 raft_consensus.cc:1273] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:22.767089 12850 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:22.906606 12860 consensus_queue.cc:1035] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:22.908466 12860 raft_consensus.cc:2953] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:22.916776 9974 raft_consensus.cc:2953] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:22.918766 7653 catalog_manager.cc:5582] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 6 to 7, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:22.928984 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 83cfc975906745a09c5e1dcfac812ae2 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:22.932106 8061 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "eed3236c4fe8455cb9527b9f4b754e81"
dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52554
I20250627 13:54:22.932273 8061 raft_consensus.cc:604] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Received request to transfer leadership
I20250627 13:54:22.932219 10102 consensus_queue.cc:237] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } }
I20250627 13:54:22.935384 10102 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:22.950842 8063 raft_consensus.cc:1273] T 5f176b66fdce4398be3492a5f3b4ea27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Refusing update from remote peer dede81a8c0164427a61532d01b6fc157: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20250627 13:54:22.954257 12844 consensus_queue.cc:1035] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:22.957726 7801 raft_consensus.cc:1273] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Refusing update from remote peer dede81a8c0164427a61532d01b6fc157: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20250627 13:54:22.961208 12844 consensus_queue.cc:1035] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:22.965256 12844 raft_consensus.cc:2953] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:22.964083 8062 raft_consensus.cc:2953] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:22.970422 7800 raft_consensus.cc:2953] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:22.970844 12757 raft_consensus.cc:991] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660: : Instructing follower 956ff058e6d64579a1963aabdb6d466b to start an election
I20250627 13:54:22.973730 12757 raft_consensus.cc:1079] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Signalling peer 956ff058e6d64579a1963aabdb6d466b to start an election
I20250627 13:54:22.974676 9840 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "eed3236c4fe8455cb9527b9f4b754e81"
dest_uuid: "956ff058e6d64579a1963aabdb6d466b"
from {username='slave'} at 127.0.103.67:41969
I20250627 13:54:22.974939 9840 raft_consensus.cc:491] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:54:22.975131 9840 raft_consensus.cc:3058] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:22.978891 9840 raft_consensus.cc:513] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:22.981541 7801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eed3236c4fe8455cb9527b9f4b754e81" candidate_uuid: "956ff058e6d64579a1963aabdb6d466b" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "83574c1bf8ea45adaff939158f93c096"
I20250627 13:54:22.981832 7801 raft_consensus.cc:3058] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:22.986129 7801 raft_consensus.cc:2466] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 956ff058e6d64579a1963aabdb6d466b in term 2.
I20250627 13:54:22.990473 9840 leader_election.cc:290] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b [CANDIDATE]: Term 2 election: Requested vote from peers 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467), 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933), 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101)
I20250627 13:54:22.998530 9711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eed3236c4fe8455cb9527b9f4b754e81" candidate_uuid: "956ff058e6d64579a1963aabdb6d466b" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "2a1282ebe92a4c9a92eed3b13f69012a"
I20250627 13:54:22.998847 9711 raft_consensus.cc:3058] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:23.000295 8063 raft_consensus.cc:2953] T 5f176b66fdce4398be3492a5f3b4ea27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:23.002238 9711 raft_consensus.cc:2466] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 956ff058e6d64579a1963aabdb6d466b in term 2.
I20250627 13:54:23.002765 7647 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 5f176b66fdce4398be3492a5f3b4ea27 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:23.002864 9778 leader_election.cc:304] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2a1282ebe92a4c9a92eed3b13f69012a, 83574c1bf8ea45adaff939158f93c096, 956ff058e6d64579a1963aabdb6d466b; no voters:
I20250627 13:54:23.003347 12854 raft_consensus.cc:2802] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:54:23.003511 12854 raft_consensus.cc:695] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b [term 2 LEADER]: Becoming Leader. State: Replica: 956ff058e6d64579a1963aabdb6d466b, State: Running, Role: LEADER
I20250627 13:54:23.004400 12854 consensus_queue.cc:237] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:23.007128 7660 catalog_manager.cc:5582] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 5 to 7, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New cstate: current_term: 2 leader_uuid: "dede81a8c0164427a61532d01b6fc157" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:23.012514 8171 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f176b66fdce4398be3492a5f3b4ea27 with delete_type TABLET_DATA_TOMBSTONED (TS e4c831df4da841929d13d6c783099b0e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:48616
I20250627 13:54:23.012971 12858 tablet_replica.cc:331] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e: stopping tablet replica
I20250627 13:54:23.013218 12858 raft_consensus.cc:2241] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:23.013423 12858 raft_consensus.cc:2270] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:23.014643 12858 ts_tablet_manager.cc:1905] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:23.022432 12858 ts_tablet_manager.cc:1918] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20250627 13:54:23.022661 12858 log.cc:1199] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal/wals/5f176b66fdce4398be3492a5f3b4ea27
I20250627 13:54:23.023582 7648 catalog_manager.cc:4928] TS e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): tablet 5f176b66fdce4398be3492a5f3b4ea27 (table rebalance_test_table_2 [id=6defd4a16efc47569b2193b71e4b2c4a]) successfully deleted
I20250627 13:54:23.032105 7653 catalog_manager.cc:5582] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b reported cstate change: term changed from 1 to 2, leader changed from 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) to 956ff058e6d64579a1963aabdb6d466b (127.0.103.70). New cstate: current_term: 2 leader_uuid: "956ff058e6d64579a1963aabdb6d466b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:23.033587 8063 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eed3236c4fe8455cb9527b9f4b754e81" candidate_uuid: "956ff058e6d64579a1963aabdb6d466b" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660"
I20250627 13:54:23.033564 8189 consensus_queue.cc:237] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:23.033776 8063 raft_consensus.cc:3053] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Stepping down as leader of term 1
I20250627 13:54:23.033861 8063 raft_consensus.cc:738] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Running, Role: LEADER
I20250627 13:54:23.034090 8063 consensus_queue.cc:260] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:23.034444 8063 raft_consensus.cc:3058] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:23.040797 8063 raft_consensus.cc:2466] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 956ff058e6d64579a1963aabdb6d466b in term 2.
I20250627 13:54:23.042924 9840 raft_consensus.cc:1273] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20250627 13:54:23.043174 8129 consensus_peers.cc:487] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e -> Peer 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101): Couldn't send request to peer 2a1282ebe92a4c9a92eed3b13f69012a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da90822ff15e4e5da05114ef38d3970a. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:23.043675 7927 raft_consensus.cc:1273] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:23.044473 12735 consensus_queue.cc:1035] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:23.044890 12735 consensus_queue.cc:1035] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:23.045739 8063 consensus_queue.cc:237] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } }
I20250627 13:54:23.070814 12836 raft_consensus.cc:2953] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:23.074566 9841 raft_consensus.cc:2953] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:23.081722 7653 catalog_manager.cc:5582] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b reported cstate change: config changed from index 8 to 9, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
W20250627 13:54:23.090018 7996 consensus_peers.cc:487] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467): Couldn't send request to peer 83574c1bf8ea45adaff939158f93c096. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:23.091068 9840 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:23.091110 7927 raft_consensus.cc:2953] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:23.091626 12757 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:23.102262 10102 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:23.102903 12757 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:23.104426 12850 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } }
I20250627 13:54:23.104956 9841 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } }
I20250627 13:54:23.112581 7653 catalog_manager.cc:5582] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 9 to 10, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:23.139935 12920 ts_tablet_manager.cc:927] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:23.141181 10104 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } }
I20250627 13:54:23.155169 12926 ts_tablet_manager.cc:927] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854: Initiating tablet copy from peer e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:54:23.157907 12926 tablet_copy_client.cc:323] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.68:46139
I20250627 13:54:23.158706 8213 tablet_copy_service.cc:140] P e4c831df4da841929d13d6c783099b0e: Received BeginTabletCopySession request for tablet e3053b529a264e8697a1df9e17eeea81 from peer a41a89ae8ed743edb2b5cf4490d8c854 ({username='slave'} at 127.0.103.66:33401)
I20250627 13:54:23.158852 8213 tablet_copy_service.cc:161] P e4c831df4da841929d13d6c783099b0e: Beginning new tablet copy session on tablet e3053b529a264e8697a1df9e17eeea81 from peer a41a89ae8ed743edb2b5cf4490d8c854 at {username='slave'} at 127.0.103.66:33401: session id = a41a89ae8ed743edb2b5cf4490d8c854-e3053b529a264e8697a1df9e17eeea81
I20250627 13:54:23.164752 8213 tablet_copy_source_session.cc:215] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:23.165575 8063 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "91664403eed74badbeba83d69f597e1f"
dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52602
I20250627 13:54:23.165803 8063 raft_consensus.cc:604] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Received request to transfer leadership
I20250627 13:54:23.169268 12920 tablet_copy_client.cc:323] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:23.169961 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet 118d5145b23f43798e1c727f2a9e9f92 from peer 956ff058e6d64579a1963aabdb6d466b ({username='slave'} at 127.0.103.70:59981)
I20250627 13:54:23.170126 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet 118d5145b23f43798e1c727f2a9e9f92 from peer 956ff058e6d64579a1963aabdb6d466b at {username='slave'} at 127.0.103.70:59981: session id = 956ff058e6d64579a1963aabdb6d466b-118d5145b23f43798e1c727f2a9e9f92
I20250627 13:54:23.173112 8082 tablet_copy_source_session.cc:215] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:23.175760 12920 ts_tablet_manager.cc:1905] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250627 13:54:23.181316 12926 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3053b529a264e8697a1df9e17eeea81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:23.186033 8063 consensus_queue.cc:237] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } }
I20250627 13:54:23.191058 9974 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:23.191134 8189 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:23.191936 12638 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20250627 13:54:23.194516 7998 consensus_peers.cc:487] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675): Couldn't send request to peer a41a89ae8ed743edb2b5cf4490d8c854. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20250627 13:54:23.195288 7996 consensus_peers.cc:487] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:23.195614 12180 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:23.197263 12638 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:23.204588 8189 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:23.209271 12920 ts_tablet_manager.cc:1918] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20250627 13:54:23.209604 12920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 118d5145b23f43798e1c727f2a9e9f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:23.212787 8063 raft_consensus.cc:1273] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20250627 13:54:23.213496 12570 consensus_queue.cc:1035] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:23.216688 9974 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:23.238917 8190 raft_consensus.cc:1273] T c2468f4746224f0ab6f4ba4b3a628e35 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20250627 13:54:23.239660 12570 consensus_queue.cc:1035] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:23.249867 7653 catalog_manager.cc:5582] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe reported cstate change: config changed from index 9 to 10, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:23.261691 9974 raft_consensus.cc:1273] T c2468f4746224f0ab6f4ba4b3a628e35 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20250627 13:54:23.262382 12570 consensus_queue.cc:1035] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250627 13:54:23.315382 7800 consensus_queue.cc:237] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:23.317030 8190 consensus_queue.cc:237] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } }
I20250627 13:54:23.318905 8189 raft_consensus.cc:1273] T c2468f4746224f0ab6f4ba4b3a628e35 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20250627 13:54:23.325798 12732 consensus_queue.cc:1035] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:23.329262 12941 raft_consensus.cc:2953] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:23.329785 8063 consensus_queue.cc:237] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } }
I20250627 13:54:23.329622 8189 raft_consensus.cc:2953] T c2468f4746224f0ab6f4ba4b3a628e35 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:23.336264 7653 catalog_manager.cc:5582] T c2468f4746224f0ab6f4ba4b3a628e35 P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 5 to 7, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New cstate: current_term: 2 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:23.322187 12926 tablet_copy_client.cc:806] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:23.338692 12926 tablet_copy_client.cc:670] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854: tablet copy: Starting download of 1 WAL segments...
W20250627 13:54:23.339754 7999 consensus_peers.cc:487] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): Couldn't send request to peer e4c831df4da841929d13d6c783099b0e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:23.340559 8041 tablet_service.cc:1515] Processing DeleteTablet for tablet c2468f4746224f0ab6f4ba4b3a628e35 with delete_type TABLET_DATA_TOMBSTONED (TS 9cf479bdcd0242738d2a77db4ad9f660 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59760
I20250627 13:54:23.340989 12926 tablet_copy_client.cc:538] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:23.341442 9841 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:23.342080 12638 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:23.348085 12926 tablet_bootstrap.cc:492] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:54:23.350256 10104 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:23.350934 12638 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:23.366645 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet c2468f4746224f0ab6f4ba4b3a628e35 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20250627 13:54:23.370834 8128 consensus_peers.cc:487] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e -> Peer a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675): Couldn't send request to peer a41a89ae8ed743edb2b5cf4490d8c854. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:23.371052 9974 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250627 13:54:23.371609 9711 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250627 13:54:23.371707 12836 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250627 13:54:23.372195 12836 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250627 13:54:23.374481 12836 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:23.379572 7648 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet e3053b529a264e8697a1df9e17eeea81 with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:23.379670 9711 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:23.382017 12920 tablet_copy_client.cc:806] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:23.381821 7653 catalog_manager.cc:5582] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 14 to 15, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:23.382280 12920 tablet_copy_client.cc:670] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:23.387833 9974 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:23.389061 9973 raft_consensus.cc:1273] T c2468f4746224f0ab6f4ba4b3a628e35 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20250627 13:54:23.390476 9822 tablet_service.cc:1515] Processing DeleteTablet for tablet e3053b529a264e8697a1df9e17eeea81 with delete_type TABLET_DATA_TOMBSTONED (TS 956ff058e6d64579a1963aabdb6d466b not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:55228
I20250627 13:54:23.394543 12941 consensus_queue.cc:1035] T c2468f4746224f0ab6f4ba4b3a628e35 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:23.396693 9711 raft_consensus.cc:1273] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 FOLLOWER]: Refusing update from remote peer 956ff058e6d64579a1963aabdb6d466b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250627 13:54:23.397383 12854 consensus_queue.cc:1035] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:23.404711 12920 tablet_copy_client.cc:538] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:23.405946 9972 raft_consensus.cc:2953] T c2468f4746224f0ab6f4ba4b3a628e35 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:23.406991 12956 ts_tablet_manager.cc:927] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854: Initiating tablet copy from peer e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:54:23.412531 7800 raft_consensus.cc:1273] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Refusing update from remote peer 956ff058e6d64579a1963aabdb6d466b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250627 13:54:23.418449 12854 consensus_queue.cc:1035] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.006s
I20250627 13:54:23.422228 12953 tablet_replica.cc:331] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660: stopping tablet replica
I20250627 13:54:23.422477 12953 raft_consensus.cc:2241] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:23.422760 12953 raft_consensus.cc:2270] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:23.425626 12848 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
W20250627 13:54:23.426873 12952 log.cc:927] Time spent T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20250627 13:54:23.430696 12953 ts_tablet_manager.cc:1905] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:23.433636 9841 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
I20250627 13:54:23.434401 10104 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
I20250627 13:54:23.441151 7657 catalog_manager.cc:5582] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b reported cstate change: config changed from index 9 to 10, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
I20250627 13:54:23.444525 12962 tablet_replica.cc:331] T e3053b529a264e8697a1df9e17eeea81 P 956ff058e6d64579a1963aabdb6d466b: stopping tablet replica
I20250627 13:54:23.448400 12956 tablet_copy_client.cc:323] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.68:46139
I20250627 13:54:23.450536 8213 tablet_copy_service.cc:140] P e4c831df4da841929d13d6c783099b0e: Received BeginTabletCopySession request for tablet 8ab02f13e45c48d293e2d2d229d11e27 from peer a41a89ae8ed743edb2b5cf4490d8c854 ({username='slave'} at 127.0.103.66:33401)
I20250627 13:54:23.450769 8213 tablet_copy_service.cc:161] P e4c831df4da841929d13d6c783099b0e: Beginning new tablet copy session on tablet 8ab02f13e45c48d293e2d2d229d11e27 from peer a41a89ae8ed743edb2b5cf4490d8c854 at {username='slave'} at 127.0.103.66:33401: session id = a41a89ae8ed743edb2b5cf4490d8c854-8ab02f13e45c48d293e2d2d229d11e27
I20250627 13:54:23.457154 8213 tablet_copy_source_session.cc:215] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:23.458281 8063 consensus_queue.cc:237] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } }
I20250627 13:54:23.459302 12848 raft_consensus.cc:991] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660: : Instructing follower a41a89ae8ed743edb2b5cf4490d8c854 to start an election
I20250627 13:54:23.459452 12848 raft_consensus.cc:1079] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Signalling peer a41a89ae8ed743edb2b5cf4490d8c854 to start an election
I20250627 13:54:23.460194 7931 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "91664403eed74badbeba83d69f597e1f"
dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
from {username='slave'} at 127.0.103.67:51721
I20250627 13:54:23.460436 7931 raft_consensus.cc:491] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:54:23.460561 7931 raft_consensus.cc:3058] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:23.463683 7931 raft_consensus.cc:513] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:23.464293 12962 raft_consensus.cc:2241] T e3053b529a264e8697a1df9e17eeea81 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:23.464740 12962 raft_consensus.cc:2270] T e3053b529a264e8697a1df9e17eeea81 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:23.465058 8062 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91664403eed74badbeba83d69f597e1f" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660"
I20250627 13:54:23.465250 8062 raft_consensus.cc:3053] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Stepping down as leader of term 1
I20250627 13:54:23.465322 8062 raft_consensus.cc:738] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Running, Role: LEADER
I20250627 13:54:23.465581 12956 ts_tablet_manager.cc:1905] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250627 13:54:23.465617 8062 consensus_queue.cc:260] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:23.465932 8062 raft_consensus.cc:3058] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:23.469281 8062 raft_consensus.cc:2466] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41a89ae8ed743edb2b5cf4490d8c854 in term 2.
I20250627 13:54:23.470889 7931 leader_election.cc:290] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 2 election: Requested vote from peers 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933), dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213), 6686c6c365bf4f4582058890161676fe (127.0.103.71:37859)
W20250627 13:54:23.473845 7996 consensus_peers.cc:487] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467): Couldn't send request to peer 83574c1bf8ea45adaff939158f93c096. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 644985e3086d4614b6598a727cb60c1d. This is attempt 1: this message will repeat every 5th retry.
W20250627 13:54:23.479132 7999 consensus_peers.cc:487] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): Couldn't send request to peer e4c831df4da841929d13d6c783099b0e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:23.480046 9841 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:23.480677 12638 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:23.483940 12638 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } }
I20250627 13:54:23.485746 10103 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91664403eed74badbeba83d69f597e1f" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "dede81a8c0164427a61532d01b6fc157"
I20250627 13:54:23.486020 10103 raft_consensus.cc:3058] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:23.487898 9840 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } }
I20250627 13:54:23.488903 10103 raft_consensus.cc:2466] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41a89ae8ed743edb2b5cf4490d8c854 in term 2.
I20250627 13:54:23.491155 10102 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:23.491485 7868 leader_election.cc:304] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 9cf479bdcd0242738d2a77db4ad9f660, a41a89ae8ed743edb2b5cf4490d8c854, dede81a8c0164427a61532d01b6fc157; no voters:
I20250627 13:54:23.491856 12115 raft_consensus.cc:2802] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:54:23.495764 12180 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:23.500480 7658 catalog_manager.cc:5582] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 10 to 11, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:23.522110 12920 tablet_bootstrap.cc:492] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap starting.
I20250627 13:54:23.495446 12962 ts_tablet_manager.cc:1905] T e3053b529a264e8697a1df9e17eeea81 P 956ff058e6d64579a1963aabdb6d466b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:23.512908 10104 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } }
I20250627 13:54:23.534186 12976 ts_tablet_manager.cc:927] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:23.547258 12953 ts_tablet_manager.cc:1918] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20250627 13:54:23.547504 12953 log.cc:1199] T c2468f4746224f0ab6f4ba4b3a628e35 P 9cf479bdcd0242738d2a77db4ad9f660: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal/wals/c2468f4746224f0ab6f4ba4b3a628e35
I20250627 13:54:23.548795 7645 catalog_manager.cc:4928] TS 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933): tablet c2468f4746224f0ab6f4ba4b3a628e35 (table rebalance_test_table_7 [id=46b91111142d4e388d01cd06d3d41940]) successfully deleted
I20250627 13:54:23.547456 8062 raft_consensus.cc:1273] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Refusing update from remote peer 956ff058e6d64579a1963aabdb6d466b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250627 13:54:23.549543 12855 consensus_queue.cc:1035] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:23.557830 7931 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "03640a5124d2405faf4a26d0e22e1ffc"
dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51356
I20250627 13:54:23.558096 7931 raft_consensus.cc:604] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Received request to transfer leadership
I20250627 13:54:23.558095 12115 raft_consensus.cc:695] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 LEADER]: Becoming Leader. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Running, Role: LEADER
I20250627 13:54:23.562203 12115 consensus_queue.cc:237] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:23.571521 9841 consensus_queue.cc:237] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:23.573648 9709 raft_consensus.cc:1273] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 FOLLOWER]: Refusing update from remote peer 956ff058e6d64579a1963aabdb6d466b: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250627 13:54:23.574486 12855 consensus_queue.cc:1035] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:23.577636 9709 raft_consensus.cc:2953] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:23.581422 12976 tablet_copy_client.cc:323] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:23.582238 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet d2eb537d49804e1b8ec5a217b1564988 from peer 83574c1bf8ea45adaff939158f93c096 ({username='slave'} at 127.0.103.65:50643)
I20250627 13:54:23.582402 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet d2eb537d49804e1b8ec5a217b1564988 from peer 83574c1bf8ea45adaff939158f93c096 at {username='slave'} at 127.0.103.65:50643: session id = 83574c1bf8ea45adaff939158f93c096-d2eb537d49804e1b8ec5a217b1564988
I20250627 13:54:23.583811 7658 catalog_manager.cc:5582] T eed3236c4fe8455cb9527b9f4b754e81 P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: config changed from index 9 to 11, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New cstate: current_term: 2 leader_uuid: "956ff058e6d64579a1963aabdb6d466b" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:23.586450 8082 tablet_copy_source_session.cc:215] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:23.589875 7800 raft_consensus.cc:1273] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Refusing update from remote peer 956ff058e6d64579a1963aabdb6d466b: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250627 13:54:23.590432 12855 consensus_queue.cc:1035] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:23.591918 7801 raft_consensus.cc:2953] T eed3236c4fe8455cb9527b9f4b754e81 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:23.594877 12962 ts_tablet_manager.cc:1918] T e3053b529a264e8697a1df9e17eeea81 P 956ff058e6d64579a1963aabdb6d466b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.14
I20250627 13:54:23.595054 12962 log.cc:1199] T e3053b529a264e8697a1df9e17eeea81 P 956ff058e6d64579a1963aabdb6d466b: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal/wals/e3053b529a264e8697a1df9e17eeea81
I20250627 13:54:23.596010 7645 catalog_manager.cc:4928] TS 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): tablet e3053b529a264e8697a1df9e17eeea81 (table rebalance_test_table_0 [id=d8bfc9268a4a4c2c92bc20d56ac73086]) successfully deleted
I20250627 13:54:23.596287 12976 ts_tablet_manager.cc:1905] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250627 13:54:23.600646 12956 ts_tablet_manager.cc:1918] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20250627 13:54:23.601009 12956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ab02f13e45c48d293e2d2d229d11e27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:23.603497 8041 tablet_service.cc:1515] Processing DeleteTablet for tablet eed3236c4fe8455cb9527b9f4b754e81 with delete_type TABLET_DATA_TOMBSTONED (TS 9cf479bdcd0242738d2a77db4ad9f660 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:59760
I20250627 13:54:23.609786 12976 ts_tablet_manager.cc:1918] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20250627 13:54:23.610219 12976 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2eb537d49804e1b8ec5a217b1564988. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:23.617377 12956 tablet_copy_client.cc:806] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:23.617669 12956 tablet_copy_client.cc:670] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:23.617494 7658 catalog_manager.cc:5582] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: term changed from 1 to 2, leader changed from 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) to a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66). New cstate: current_term: 2 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:23.621228 12956 tablet_copy_client.cc:538] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:23.621178 12854 raft_consensus.cc:2953] T eed3236c4fe8455cb9527b9f4b754e81 P 956ff058e6d64579a1963aabdb6d466b [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:23.627612 12953 tablet_replica.cc:331] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660: stopping tablet replica
I20250627 13:54:23.627815 12953 raft_consensus.cc:2241] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:23.627996 12953 raft_consensus.cc:2270] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:23.628408 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet eed3236c4fe8455cb9527b9f4b754e81 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:23.631458 12953 ts_tablet_manager.cc:1905] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:23.632061 12956 tablet_bootstrap.cc:492] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:54:23.642505 12953 ts_tablet_manager.cc:1918] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20250627 13:54:23.644892 12976 tablet_copy_client.cc:806] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:23.646142 12976 tablet_copy_client.cc:670] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:23.648856 12976 tablet_copy_client.cc:538] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:23.651386 12953 log.cc:1199] T eed3236c4fe8455cb9527b9f4b754e81 P 9cf479bdcd0242738d2a77db4ad9f660: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal/wals/eed3236c4fe8455cb9527b9f4b754e81
I20250627 13:54:23.655670 7645 catalog_manager.cc:4928] TS 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933): tablet eed3236c4fe8455cb9527b9f4b754e81 (table rebalance_test_table_9 [id=19e5686b50fe4187a733aa6413ef2ce0]) successfully deleted
I20250627 13:54:23.664826 12975 raft_consensus.cc:991] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854: : Instructing follower e4c831df4da841929d13d6c783099b0e to start an election
I20250627 13:54:23.665036 12975 raft_consensus.cc:1079] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Signalling peer e4c831df4da841929d13d6c783099b0e to start an election
I20250627 13:54:23.667516 9972 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91664403eed74badbeba83d69f597e1f" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "6686c6c365bf4f4582058890161676fe"
I20250627 13:54:23.667753 9972 raft_consensus.cc:3058] T 91664403eed74badbeba83d69f597e1f P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:23.670989 9972 raft_consensus.cc:2466] T 91664403eed74badbeba83d69f597e1f P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41a89ae8ed743edb2b5cf4490d8c854 in term 2.
I20250627 13:54:23.678798 8190 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "03640a5124d2405faf4a26d0e22e1ffc"
dest_uuid: "e4c831df4da841929d13d6c783099b0e"
from {username='slave'} at 127.0.103.66:33401
I20250627 13:54:23.679059 8190 raft_consensus.cc:491] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:54:23.679183 8190 raft_consensus.cc:3058] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:23.682168 8190 raft_consensus.cc:513] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } }
I20250627 13:54:23.683696 7931 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03640a5124d2405faf4a26d0e22e1ffc" candidate_uuid: "e4c831df4da841929d13d6c783099b0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
I20250627 13:54:23.683902 7931 raft_consensus.cc:3053] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Stepping down as leader of term 1
I20250627 13:54:23.683990 7931 raft_consensus.cc:738] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Running, Role: LEADER
I20250627 13:54:23.684329 7931 consensus_queue.cc:260] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } }
I20250627 13:54:23.684777 7931 raft_consensus.cc:3058] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:23.687705 7931 raft_consensus.cc:2466] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c831df4da841929d13d6c783099b0e in term 2.
I20250627 13:54:23.690217 8190 leader_election.cc:290] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [CANDIDATE]: Term 2 election: Requested vote from peers a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675), 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647), dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213)
I20250627 13:54:23.690744 10104 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03640a5124d2405faf4a26d0e22e1ffc" candidate_uuid: "e4c831df4da841929d13d6c783099b0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "dede81a8c0164427a61532d01b6fc157"
I20250627 13:54:23.690929 10104 raft_consensus.cc:3058] T 03640a5124d2405faf4a26d0e22e1ffc P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:23.694664 10104 raft_consensus.cc:2466] T 03640a5124d2405faf4a26d0e22e1ffc P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c831df4da841929d13d6c783099b0e in term 2.
I20250627 13:54:23.695104 8128 leader_election.cc:304] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: a41a89ae8ed743edb2b5cf4490d8c854, dede81a8c0164427a61532d01b6fc157, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:54:23.695668 12836 raft_consensus.cc:2802] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:54:23.695804 12836 raft_consensus.cc:695] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 2 LEADER]: Becoming Leader. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Running, Role: LEADER
I20250627 13:54:23.695809 12976 tablet_bootstrap.cc:492] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:54:23.696089 12836 consensus_queue.cc:237] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } }
I20250627 13:54:23.711773 13005 ts_tablet_manager.cc:927] T da90822ff15e4e5da05114ef38d3970a P 2a1282ebe92a4c9a92eed3b13f69012a: Initiating tablet copy from peer e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:54:23.711731 9841 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03640a5124d2405faf4a26d0e22e1ffc" candidate_uuid: "e4c831df4da841929d13d6c783099b0e" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "956ff058e6d64579a1963aabdb6d466b"
I20250627 13:54:23.712167 9841 raft_consensus.cc:3058] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:23.721290 13005 tablet_copy_client.cc:323] T da90822ff15e4e5da05114ef38d3970a P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.68:46139
I20250627 13:54:23.722115 8213 tablet_copy_service.cc:140] P e4c831df4da841929d13d6c783099b0e: Received BeginTabletCopySession request for tablet da90822ff15e4e5da05114ef38d3970a from peer 2a1282ebe92a4c9a92eed3b13f69012a ({username='slave'} at 127.0.103.69:43551)
I20250627 13:54:23.722287 8213 tablet_copy_service.cc:161] P e4c831df4da841929d13d6c783099b0e: Beginning new tablet copy session on tablet da90822ff15e4e5da05114ef38d3970a from peer 2a1282ebe92a4c9a92eed3b13f69012a at {username='slave'} at 127.0.103.69:43551: session id = 2a1282ebe92a4c9a92eed3b13f69012a-da90822ff15e4e5da05114ef38d3970a
I20250627 13:54:23.725073 9841 raft_consensus.cc:2466] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c831df4da841929d13d6c783099b0e in term 2.
I20250627 13:54:23.725473 8213 tablet_copy_source_session.cc:215] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:23.726929 13005 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da90822ff15e4e5da05114ef38d3970a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:23.745553 12926 tablet_bootstrap.cc:492] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:23.746057 12926 tablet_bootstrap.cc:492] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap complete.
I20250627 13:54:23.746368 12926 ts_tablet_manager.cc:1397] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.398s user 0.072s sys 0.010s
I20250627 13:54:23.747167 12926 raft_consensus.cc:357] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } }
I20250627 13:54:23.747404 12926 raft_consensus.cc:738] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: LEARNER
I20250627 13:54:23.751678 13022 ts_tablet_manager.cc:927] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:23.752724 13022 tablet_copy_client.cc:323] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:23.753399 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet 118d5145b23f43798e1c727f2a9e9f92 from peer a41a89ae8ed743edb2b5cf4490d8c854 ({username='slave'} at 127.0.103.66:40009)
I20250627 13:54:23.753582 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet 118d5145b23f43798e1c727f2a9e9f92 from peer a41a89ae8ed743edb2b5cf4490d8c854 at {username='slave'} at 127.0.103.66:40009: session id = a41a89ae8ed743edb2b5cf4490d8c854-118d5145b23f43798e1c727f2a9e9f92
I20250627 13:54:23.756453 8082 tablet_copy_source_session.cc:215] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:23.758074 13022 ts_tablet_manager.cc:1905] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250627 13:54:23.769016 7653 catalog_manager.cc:5582] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e reported cstate change: term changed from 1 to 2, leader changed from a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) to e4c831df4da841929d13d6c783099b0e (127.0.103.68). New cstate: current_term: 2 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:23.776171 7800 consensus_queue.cc:237] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
W20250627 13:54:23.786839 7738 consensus_peers.cc:487] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 -> Peer 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101): Couldn't send request to peer 2a1282ebe92a4c9a92eed3b13f69012a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40431e3bd7244904a4e65f59477bd21d. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:23.787097 9972 raft_consensus.cc:1273] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:23.787629 12732 consensus_queue.cc:1035] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:23.792102 7931 raft_consensus.cc:1273] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250627 13:54:23.792773 12732 consensus_queue.cc:1035] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250627 13:54:23.794029 12920 tablet_bootstrap.cc:492] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:23.794517 12920 tablet_bootstrap.cc:492] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap complete.
I20250627 13:54:23.794844 12920 ts_tablet_manager.cc:1397] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: Time spent bootstrapping tablet: real 0.273s user 0.063s sys 0.045s
I20250627 13:54:23.795796 12920 raft_consensus.cc:357] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:23.796082 12920 raft_consensus.cc:738] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 956ff058e6d64579a1963aabdb6d466b, State: Initialized, Role: LEARNER
I20250627 13:54:23.796337 12920 consensus_queue.cc:260] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:23.799378 13005 tablet_copy_client.cc:806] T da90822ff15e4e5da05114ef38d3970a P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:23.799659 13005 tablet_copy_client.cc:670] T da90822ff15e4e5da05114ef38d3970a P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:23.747612 12926 consensus_queue.cc:260] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 1.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } }
I20250627 13:54:23.803963 12926 ts_tablet_manager.cc:1428] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.057s user 0.002s sys 0.000s
I20250627 13:54:23.804595 8213 tablet_copy_service.cc:342] P e4c831df4da841929d13d6c783099b0e: Request end of tablet copy session a41a89ae8ed743edb2b5cf4490d8c854-e3053b529a264e8697a1df9e17eeea81 received from {username='slave'} at 127.0.103.66:33401
I20250627 13:54:23.804752 8213 tablet_copy_service.cc:434] P e4c831df4da841929d13d6c783099b0e: ending tablet copy session a41a89ae8ed743edb2b5cf4490d8c854-e3053b529a264e8697a1df9e17eeea81 on tablet e3053b529a264e8697a1df9e17eeea81 with peer a41a89ae8ed743edb2b5cf4490d8c854
I20250627 13:54:23.805485 13005 tablet_copy_client.cc:538] T da90822ff15e4e5da05114ef38d3970a P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:23.809478 13005 tablet_bootstrap.cc:492] T da90822ff15e4e5da05114ef38d3970a P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap starting.
I20250627 13:54:23.810083 12920 ts_tablet_manager.cc:1428] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b: Time spent starting tablet: real 0.015s user 0.000s sys 0.003s
I20250627 13:54:23.810652 12732 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:23.813992 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session 956ff058e6d64579a1963aabdb6d466b-118d5145b23f43798e1c727f2a9e9f92 received from {username='slave'} at 127.0.103.70:59981
I20250627 13:54:23.814133 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session 956ff058e6d64579a1963aabdb6d466b-118d5145b23f43798e1c727f2a9e9f92 on tablet 118d5145b23f43798e1c727f2a9e9f92 with peer 956ff058e6d64579a1963aabdb6d466b
I20250627 13:54:23.815558 9973 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:23.823925 7653 catalog_manager.cc:5582] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe reported cstate change: config changed from index 6 to 7, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:23.826601 13022 ts_tablet_manager.cc:1918] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20250627 13:54:23.826931 13022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 118d5145b23f43798e1c727f2a9e9f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:23.843844 13022 tablet_copy_client.cc:806] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:23.844095 13022 tablet_copy_client.cc:670] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:23.848459 13022 tablet_copy_client.cc:538] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:23.849064 7927 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } } }
I20250627 13:54:23.851485 12976 tablet_bootstrap.cc:492] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:23.852089 12976 tablet_bootstrap.cc:492] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: Bootstrap complete.
I20250627 13:54:23.852383 12976 ts_tablet_manager.cc:1397] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.157s user 0.056s sys 0.004s
I20250627 13:54:23.853220 13022 tablet_bootstrap.cc:492] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:54:23.853264 12976 raft_consensus.cc:357] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } }
I20250627 13:54:23.853608 12976 raft_consensus.cc:738] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: LEARNER
I20250627 13:54:23.853861 12976 consensus_queue.cc:260] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } }
I20250627 13:54:23.856760 12976 ts_tablet_manager.cc:1428] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250627 13:54:23.860993 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session 83574c1bf8ea45adaff939158f93c096-d2eb537d49804e1b8ec5a217b1564988 received from {username='slave'} at 127.0.103.65:50643
I20250627 13:54:23.861145 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session 83574c1bf8ea45adaff939158f93c096-d2eb537d49804e1b8ec5a217b1564988 on tablet d2eb537d49804e1b8ec5a217b1564988 with peer 83574c1bf8ea45adaff939158f93c096
I20250627 13:54:23.907101 12956 tablet_bootstrap.cc:492] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:23.914093 12956 tablet_bootstrap.cc:492] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap complete.
I20250627 13:54:23.914461 12956 ts_tablet_manager.cc:1397] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.283s user 0.047s sys 0.011s
I20250627 13:54:23.916028 12956 raft_consensus.cc:357] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } }
I20250627 13:54:23.918437 12956 raft_consensus.cc:738] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: LEARNER
I20250627 13:54:23.918790 12956 consensus_queue.cc:260] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } }
I20250627 13:54:23.919685 12956 ts_tablet_manager.cc:1428] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250627 13:54:23.924424 8213 tablet_copy_service.cc:342] P e4c831df4da841929d13d6c783099b0e: Request end of tablet copy session a41a89ae8ed743edb2b5cf4490d8c854-8ab02f13e45c48d293e2d2d229d11e27 received from {username='slave'} at 127.0.103.66:33401
I20250627 13:54:23.924587 8213 tablet_copy_service.cc:434] P e4c831df4da841929d13d6c783099b0e: ending tablet copy session a41a89ae8ed743edb2b5cf4490d8c854-8ab02f13e45c48d293e2d2d229d11e27 on tablet 8ab02f13e45c48d293e2d2d229d11e27 with peer a41a89ae8ed743edb2b5cf4490d8c854
I20250627 13:54:23.943153 7927 raft_consensus.cc:1215] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250627 13:54:23.949331 8190 consensus_queue.cc:237] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
W20250627 13:54:23.959836 8126 consensus_peers.cc:487] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ab02f13e45c48d293e2d2d229d11e27. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:23.960878 7927 raft_consensus.cc:1273] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:23.961694 12860 consensus_queue.cc:1035] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:23.965952 9972 raft_consensus.cc:1273] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:23.966820 12860 consensus_queue.cc:1035] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:23.981937 8063 raft_consensus.cc:1273] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:23.982690 12860 consensus_queue.cc:1035] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:23.985481 12836 raft_consensus.cc:2953] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:23.990706 8063 raft_consensus.cc:2953] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:23.999411 7658 catalog_manager.cc:5582] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 7 to 8, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:24.008433 8063 raft_consensus.cc:1273] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250627 13:54:24.009148 12975 consensus_queue.cc:1035] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:24.012773 7800 raft_consensus.cc:1215] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20250627 13:54:24.025163 13005 tablet_bootstrap.cc:492] T da90822ff15e4e5da05114ef38d3970a P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:24.025334 10104 raft_consensus.cc:1273] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250627 13:54:24.025924 9973 raft_consensus.cc:1273] T 91664403eed74badbeba83d69f597e1f P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250627 13:54:24.025955 12975 consensus_queue.cc:1035] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:24.026553 13005 tablet_bootstrap.cc:492] T da90822ff15e4e5da05114ef38d3970a P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap complete.
I20250627 13:54:24.026576 12975 consensus_queue.cc:1035] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:24.029493 13005 ts_tablet_manager.cc:1397] T da90822ff15e4e5da05114ef38d3970a P 2a1282ebe92a4c9a92eed3b13f69012a: Time spent bootstrapping tablet: real 0.220s user 0.061s sys 0.009s
I20250627 13:54:24.030683 13005 raft_consensus.cc:357] T da90822ff15e4e5da05114ef38d3970a P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:24.032339 13005 raft_consensus.cc:738] T da90822ff15e4e5da05114ef38d3970a P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a1282ebe92a4c9a92eed3b13f69012a, State: Initialized, Role: LEARNER
I20250627 13:54:24.032816 13005 consensus_queue.cc:260] T da90822ff15e4e5da05114ef38d3970a P 2a1282ebe92a4c9a92eed3b13f69012a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:24.034193 13005 ts_tablet_manager.cc:1428] T da90822ff15e4e5da05114ef38d3970a P 2a1282ebe92a4c9a92eed3b13f69012a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250627 13:54:24.035286 8213 tablet_copy_service.cc:342] P e4c831df4da841929d13d6c783099b0e: Request end of tablet copy session 2a1282ebe92a4c9a92eed3b13f69012a-da90822ff15e4e5da05114ef38d3970a received from {username='slave'} at 127.0.103.69:43551
I20250627 13:54:24.035444 8213 tablet_copy_service.cc:434] P e4c831df4da841929d13d6c783099b0e: ending tablet copy session 2a1282ebe92a4c9a92eed3b13f69012a-da90822ff15e4e5da05114ef38d3970a on tablet da90822ff15e4e5da05114ef38d3970a with peer 2a1282ebe92a4c9a92eed3b13f69012a
I20250627 13:54:24.041030 12976 ts_tablet_manager.cc:927] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:24.093966 7931 consensus_queue.cc:237] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:24.094930 12976 tablet_copy_client.cc:323] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:24.095988 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet 644985e3086d4614b6598a727cb60c1d from peer 83574c1bf8ea45adaff939158f93c096 ({username='slave'} at 127.0.103.65:50643)
I20250627 13:54:24.096197 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet 644985e3086d4614b6598a727cb60c1d from peer 83574c1bf8ea45adaff939158f93c096 at {username='slave'} at 127.0.103.65:50643: session id = 83574c1bf8ea45adaff939158f93c096-644985e3086d4614b6598a727cb60c1d
I20250627 13:54:24.100021 8082 tablet_copy_source_session.cc:215] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:24.103525 9974 raft_consensus.cc:1273] T 91664403eed74badbeba83d69f597e1f P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250627 13:54:24.104528 12115 consensus_queue.cc:1035] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:24.105677 12860 raft_consensus.cc:1062] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e: attempting to promote NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 to VOTER
I20250627 13:54:24.106513 12860 consensus_queue.cc:237] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:24.106982 12976 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 644985e3086d4614b6598a727cb60c1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:24.111817 13074 ts_tablet_manager.cc:927] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:24.112860 10102 raft_consensus.cc:1273] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250627 13:54:24.113507 9841 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
W20250627 13:54:24.114790 8126 consensus_peers.cc:487] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ab02f13e45c48d293e2d2d229d11e27. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:24.115872 12975 consensus_queue.cc:1035] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:24.117061 7931 raft_consensus.cc:1273] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:24.117954 13084 consensus_queue.cc:1035] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:24.118176 12115 raft_consensus.cc:2953] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:24.117070 9974 raft_consensus.cc:2953] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:24.119351 7931 raft_consensus.cc:2953] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:24.123126 7647 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 91664403eed74badbeba83d69f597e1f with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:24.124701 10104 raft_consensus.cc:2953] T 91664403eed74badbeba83d69f597e1f P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:24.126535 7653 catalog_manager.cc:5582] T 91664403eed74badbeba83d69f597e1f P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index 9 to 11, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New cstate: current_term: 2 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:24.131521 9840 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250627 13:54:24.133140 13084 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:24.132580 8041 tablet_service.cc:1515] Processing DeleteTablet for tablet 91664403eed74badbeba83d69f597e1f with delete_type TABLET_DATA_TOMBSTONED (TS 9cf479bdcd0242738d2a77db4ad9f660 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:59760
I20250627 13:54:24.133677 12953 tablet_replica.cc:331] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660: stopping tablet replica
I20250627 13:54:24.133939 12953 raft_consensus.cc:2241] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:24.134130 12953 pending_rounds.cc:70] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660: Trying to abort 1 pending ops.
I20250627 13:54:24.134305 12953 pending_rounds.cc:77] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660: Aborting op as it isn't in flight: id { term: 2 index: 10 } timestamp: 7172228970860003328 op_type: NO_OP noop_request { }
I20250627 13:54:24.134456 12953 raft_consensus.cc:2887] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250627 13:54:24.134580 12953 raft_consensus.cc:2270] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:24.138833 12953 ts_tablet_manager.cc:1905] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:24.147961 9972 raft_consensus.cc:1273] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:24.149183 13083 consensus_queue.cc:1035] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:24.149979 12953 ts_tablet_manager.cc:1918] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20250627 13:54:24.150187 12953 log.cc:1199] T 91664403eed74badbeba83d69f597e1f P 9cf479bdcd0242738d2a77db4ad9f660: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal/wals/91664403eed74badbeba83d69f597e1f
I20250627 13:54:24.150960 8062 raft_consensus.cc:1273] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:24.151364 7645 catalog_manager.cc:4928] TS 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933): tablet 91664403eed74badbeba83d69f597e1f (table rebalance_test_table_7 [id=46b91111142d4e388d01cd06d3d41940]) successfully deleted
I20250627 13:54:24.152045 12860 consensus_queue.cc:1035] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:24.152292 12976 tablet_copy_client.cc:806] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:24.152534 12976 tablet_copy_client.cc:670] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:24.152918 13084 raft_consensus.cc:2953] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:24.153692 9974 raft_consensus.cc:2953] T 91664403eed74badbeba83d69f597e1f P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:24.156801 8062 raft_consensus.cc:2953] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:24.163028 7927 raft_consensus.cc:2953] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:24.161690 9973 raft_consensus.cc:2953] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:24.170879 7653 catalog_manager.cc:5582] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 8 to 9, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:24.172894 7933 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250627 13:54:24.173141 12976 tablet_copy_client.cc:538] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:24.173516 12860 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:24.177476 12976 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:54:24.181191 13074 tablet_copy_client.cc:323] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:24.182008 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet 644985e3086d4614b6598a727cb60c1d from peer e4c831df4da841929d13d6c783099b0e ({username='slave'} at 127.0.103.68:59937)
I20250627 13:54:24.182185 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet 644985e3086d4614b6598a727cb60c1d from peer e4c831df4da841929d13d6c783099b0e at {username='slave'} at 127.0.103.68:59937: session id = e4c831df4da841929d13d6c783099b0e-644985e3086d4614b6598a727cb60c1d
I20250627 13:54:24.185127 8082 tablet_copy_source_session.cc:215] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:24.188130 8190 consensus_queue.cc:237] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } }
I20250627 13:54:24.190946 9974 raft_consensus.cc:1273] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:24.192935 13088 consensus_queue.cc:1035] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:24.195735 13005 ts_tablet_manager.cc:927] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a: Initiating tablet copy from peer 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:54:24.196884 7931 raft_consensus.cc:1273] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:24.198173 12860 consensus_queue.cc:1035] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:24.200843 13088 raft_consensus.cc:2953] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:24.207360 10104 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250627 13:54:24.207924 12836 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:24.207964 9974 raft_consensus.cc:2953] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:24.210464 7648 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 8ab02f13e45c48d293e2d2d229d11e27 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:24.213131 7933 raft_consensus.cc:2953] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:24.213968 7658 catalog_manager.cc:5582] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe reported cstate change: config changed from index 9 to 10, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:24.218536 9822 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ab02f13e45c48d293e2d2d229d11e27 with delete_type TABLET_DATA_TOMBSTONED (TS 956ff058e6d64579a1963aabdb6d466b not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55228
I20250627 13:54:24.219610 13074 ts_tablet_manager.cc:1905] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250627 13:54:24.223302 8190 consensus_queue.cc:237] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } }
I20250627 13:54:24.224125 7927 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:24.226217 9973 raft_consensus.cc:1273] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:24.227517 7933 raft_consensus.cc:1273] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:24.228340 12860 consensus_queue.cc:1035] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:24.232189 13084 consensus_queue.cc:1035] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:24.233170 13084 raft_consensus.cc:2953] T 8ab02f13e45c48d293e2d2d229d11e27 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:24.237190 7648 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 8ab02f13e45c48d293e2d2d229d11e27 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20250627 13:54:24.239562 7645 catalog_manager.cc:4895] TS 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): delete failed for tablet 8ab02f13e45c48d293e2d2d229d11e27 because the tablet was not found. No further retry: Not found: Tablet not found: 8ab02f13e45c48d293e2d2d229d11e27
I20250627 13:54:24.239380 9973 raft_consensus.cc:2953] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:24.239777 13005 tablet_copy_client.cc:323] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:37467
I20250627 13:54:24.244043 7822 tablet_copy_service.cc:140] P 83574c1bf8ea45adaff939158f93c096: Received BeginTabletCopySession request for tablet 40431e3bd7244904a4e65f59477bd21d from peer 2a1282ebe92a4c9a92eed3b13f69012a ({username='slave'} at 127.0.103.69:43835)
I20250627 13:54:24.244207 7822 tablet_copy_service.cc:161] P 83574c1bf8ea45adaff939158f93c096: Beginning new tablet copy session on tablet 40431e3bd7244904a4e65f59477bd21d from peer 2a1282ebe92a4c9a92eed3b13f69012a at {username='slave'} at 127.0.103.69:43835: session id = 2a1282ebe92a4c9a92eed3b13f69012a-40431e3bd7244904a4e65f59477bd21d
I20250627 13:54:24.237162 7931 raft_consensus.cc:2953] T 8ab02f13e45c48d293e2d2d229d11e27 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:24.245900 7658 catalog_manager.cc:5582] T 8ab02f13e45c48d293e2d2d229d11e27 P 6686c6c365bf4f4582058890161676fe reported cstate change: config changed from index 10 to 11, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:24.247205 7822 tablet_copy_source_session.cc:215] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:24.250669 13005 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40431e3bd7244904a4e65f59477bd21d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:24.251083 8041 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ab02f13e45c48d293e2d2d229d11e27 with delete_type TABLET_DATA_TOMBSTONED (TS 9cf479bdcd0242738d2a77db4ad9f660 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:59760
I20250627 13:54:24.258147 13005 tablet_copy_client.cc:806] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:24.258452 13005 tablet_copy_client.cc:670] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:24.264587 13005 tablet_copy_client.cc:538] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:24.268173 13005 tablet_bootstrap.cc:492] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap starting.
I20250627 13:54:24.274075 13022 tablet_bootstrap.cc:492] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:24.274514 13022 tablet_bootstrap.cc:492] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap complete.
I20250627 13:54:24.274771 13022 ts_tablet_manager.cc:1397] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.422s user 0.076s sys 0.045s
I20250627 13:54:24.280059 13022 raft_consensus.cc:357] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } }
I20250627 13:54:24.280293 13022 raft_consensus.cc:738] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: LEARNER
I20250627 13:54:24.280483 13022 consensus_queue.cc:260] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } }
I20250627 13:54:24.280948 13022 ts_tablet_manager.cc:1428] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250627 13:54:24.282356 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session a41a89ae8ed743edb2b5cf4490d8c854-118d5145b23f43798e1c727f2a9e9f92 received from {username='slave'} at 127.0.103.66:40009
I20250627 13:54:24.282522 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session a41a89ae8ed743edb2b5cf4490d8c854-118d5145b23f43798e1c727f2a9e9f92 on tablet 118d5145b23f43798e1c727f2a9e9f92 with peer a41a89ae8ed743edb2b5cf4490d8c854
I20250627 13:54:24.287384 12976 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:24.287920 12976 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096: Bootstrap complete.
I20250627 13:54:24.288244 12976 ts_tablet_manager.cc:1397] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.111s user 0.056s sys 0.004s
I20250627 13:54:24.288317 9709 raft_consensus.cc:1215] T da90822ff15e4e5da05114ef38d3970a P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20250627 13:54:24.289961 12976 raft_consensus.cc:357] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } }
I20250627 13:54:24.290269 12976 raft_consensus.cc:738] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: LEARNER
I20250627 13:54:24.290537 12976 consensus_queue.cc:260] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } }
I20250627 13:54:24.291198 12976 ts_tablet_manager.cc:1428] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20250627 13:54:24.292878 7653 catalog_manager.cc:5129] ChangeConfig:REMOVE_PEER RPC for tablet 8ab02f13e45c48d293e2d2d229d11e27 with cas_config_opid_index 10: aborting the task: latest config opid_index 11; task opid_index 10
I20250627 13:54:24.293646 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session 83574c1bf8ea45adaff939158f93c096-644985e3086d4614b6598a727cb60c1d received from {username='slave'} at 127.0.103.65:50643
I20250627 13:54:24.293790 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session 83574c1bf8ea45adaff939158f93c096-644985e3086d4614b6598a727cb60c1d on tablet 644985e3086d4614b6598a727cb60c1d with peer 83574c1bf8ea45adaff939158f93c096
I20250627 13:54:24.315476 13074 ts_tablet_manager.cc:1918] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20250627 13:54:24.315865 13074 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 644985e3086d4614b6598a727cb60c1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:24.320851 12953 tablet_replica.cc:331] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660: stopping tablet replica
I20250627 13:54:24.321072 12953 raft_consensus.cc:2241] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:24.321390 12953 raft_consensus.cc:2270] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:24.322484 12953 ts_tablet_manager.cc:1905] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:24.326192 13074 tablet_copy_client.cc:806] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:24.326453 13074 tablet_copy_client.cc:670] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:24.330332 12953 ts_tablet_manager.cc:1918] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20250627 13:54:24.330463 12953 log.cc:1199] T 8ab02f13e45c48d293e2d2d229d11e27 P 9cf479bdcd0242738d2a77db4ad9f660: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal/wals/8ab02f13e45c48d293e2d2d229d11e27
I20250627 13:54:24.331122 7645 catalog_manager.cc:4928] TS 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933): tablet 8ab02f13e45c48d293e2d2d229d11e27 (table rebalance_test_table_5 [id=dd79cdfc35164ae6a649d3a77ee397c2]) successfully deleted
I20250627 13:54:24.331485 13074 tablet_copy_client.cc:538] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:24.333057 8041 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ab02f13e45c48d293e2d2d229d11e27 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 9 (current committed config index is 11)) from {username='slave'} at 127.0.0.1:59760
I20250627 13:54:24.334046 7645 catalog_manager.cc:4928] TS 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933): tablet 8ab02f13e45c48d293e2d2d229d11e27 (table rebalance_test_table_5 [id=dd79cdfc35164ae6a649d3a77ee397c2]) successfully deleted
I20250627 13:54:24.393385 13074 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
I20250627 13:54:24.430960 13005 tablet_bootstrap.cc:492] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:24.432902 13005 tablet_bootstrap.cc:492] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a: Bootstrap complete.
I20250627 13:54:24.433925 13005 ts_tablet_manager.cc:1397] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a: Time spent bootstrapping tablet: real 0.166s user 0.055s sys 0.029s
I20250627 13:54:24.448670 13005 raft_consensus.cc:357] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:24.450448 13005 raft_consensus.cc:738] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a1282ebe92a4c9a92eed3b13f69012a, State: Initialized, Role: LEARNER
I20250627 13:54:24.450908 13005 consensus_queue.cc:260] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: true } }
I20250627 13:54:24.455437 13005 ts_tablet_manager.cc:1428] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20250627 13:54:24.458587 7822 tablet_copy_service.cc:342] P 83574c1bf8ea45adaff939158f93c096: Request end of tablet copy session 2a1282ebe92a4c9a92eed3b13f69012a-40431e3bd7244904a4e65f59477bd21d received from {username='slave'} at 127.0.103.69:43835
I20250627 13:54:24.458748 7822 tablet_copy_service.cc:434] P 83574c1bf8ea45adaff939158f93c096: ending tablet copy session 2a1282ebe92a4c9a92eed3b13f69012a-40431e3bd7244904a4e65f59477bd21d on tablet 40431e3bd7244904a4e65f59477bd21d with peer 2a1282ebe92a4c9a92eed3b13f69012a
I20250627 13:54:24.497830 7800 raft_consensus.cc:1215] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20250627 13:54:24.592341 12848 raft_consensus.cc:1062] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER 83574c1bf8ea45adaff939158f93c096 to VOTER
I20250627 13:54:24.593479 12848 consensus_queue.cc:237] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } }
I20250627 13:54:24.606168 12180 raft_consensus.cc:1062] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER 956ff058e6d64579a1963aabdb6d466b to VOTER
I20250627 13:54:24.607052 12180 consensus_queue.cc:237] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } }
I20250627 13:54:24.627095 13074 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:24.627524 13074 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: Bootstrap complete.
I20250627 13:54:24.627785 13074 ts_tablet_manager.cc:1397] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.234s user 0.059s sys 0.011s
I20250627 13:54:24.628504 13074 raft_consensus.cc:357] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } }
I20250627 13:54:24.628713 13074 raft_consensus.cc:738] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: LEARNER
I20250627 13:54:24.628906 13074 consensus_queue.cc:260] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } }
I20250627 13:54:24.629745 13074 ts_tablet_manager.cc:1428] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250627 13:54:24.630610 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session e4c831df4da841929d13d6c783099b0e-644985e3086d4614b6598a727cb60c1d received from {username='slave'} at 127.0.103.68:59937
I20250627 13:54:24.630759 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session e4c831df4da841929d13d6c783099b0e-644985e3086d4614b6598a727cb60c1d on tablet 644985e3086d4614b6598a727cb60c1d with peer e4c831df4da841929d13d6c783099b0e
I20250627 13:54:24.641269 7800 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:24.642025 12638 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:24.652740 7931 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:24.652971 9841 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:24.653342 12638 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:24.653846 12638 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:24.658308 9709 raft_consensus.cc:1215] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250627 13:54:24.665093 9973 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:24.666138 13188 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20250627 13:54:24.671217 9841 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:24.672423 12638 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:24.690655 12638 raft_consensus.cc:1025] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: attempt to promote peer a41a89ae8ed743edb2b5cf4490d8c854: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250627 13:54:24.693353 12638 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:24.698998 12836 raft_consensus.cc:1062] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: attempting to promote NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 to VOTER
I20250627 13:54:24.699795 12836 consensus_queue.cc:237] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } }
I20250627 13:54:24.701486 9973 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:24.702752 10102 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:24.705027 13191 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.002s
I20250627 13:54:24.709151 9841 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:24.711612 8190 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:24.713256 13190 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:24.712481 13188 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:24.727698 7931 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20250627 13:54:24.728201 13083 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250627 13:54:24.731860 9974 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20250627 13:54:24.732661 9709 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20250627 13:54:24.733376 13083 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250627 13:54:24.733846 13083 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250627 13:54:24.735281 10104 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:24.738687 7658 catalog_manager.cc:5582] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe reported cstate change: config changed from index 10 to 11, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:24.738761 8190 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:24.742427 7653 catalog_manager.cc:5582] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 reported cstate change: config changed from index 10 to 11, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:24.742875 7931 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:24.746925 9840 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:24.755015 13083 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:24.756642 13191 raft_consensus.cc:1062] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 to VOTER
I20250627 13:54:24.758842 7800 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:24.762210 9709 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:24.762583 7931 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:24.769155 7660 catalog_manager.cc:5582] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index 15 to 16, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:24.757763 13191 consensus_queue.cc:237] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } }
I20250627 13:54:24.774838 9974 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:24.777318 8189 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250627 13:54:24.778028 9972 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250627 13:54:24.778896 13190 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:24.779436 13190 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:24.780171 7931 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250627 13:54:24.780861 12180 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:24.789628 9841 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250627 13:54:24.790511 12180 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:24.792816 8063 consensus_queue.cc:237] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } }
I20250627 13:54:24.794806 10102 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250627 13:54:24.795313 12180 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:24.796166 7800 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250627 13:54:24.797907 12180 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:24.798619 12180 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:24.801012 12848 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:24.807180 10104 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:24.815968 9841 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:24.817693 9972 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:24.823335 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet d2eb537d49804e1b8ec5a217b1564988 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:24.824159 7931 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:24.823252 7653 catalog_manager.cc:5582] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 11 to 12, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:24.825232 7655 catalog_manager.cc:5582] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b reported cstate change: config changed from index 11 to 12, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:24.828454 7800 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:24.829504 8193 consensus_queue.cc:237] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } }
I20250627 13:54:24.832255 9840 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250627 13:54:24.832254 10102 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250627 13:54:24.832808 13088 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250627 13:54:24.835059 13084 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250627 13:54:24.835345 9822 tablet_service.cc:1515] Processing DeleteTablet for tablet d2eb537d49804e1b8ec5a217b1564988 with delete_type TABLET_DATA_TOMBSTONED (TS 956ff058e6d64579a1963aabdb6d466b not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:55228
I20250627 13:54:24.835840 12836 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:24.836462 9839 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:24.840509 7648 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 03640a5124d2405faf4a26d0e22e1ffc with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:24.848137 10102 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:24.855835 13222 tablet_replica.cc:331] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b: stopping tablet replica
I20250627 13:54:24.856168 13222 raft_consensus.cc:2241] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:24.856103 7656 catalog_manager.cc:5582] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b reported cstate change: config changed from index 11 to 13, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New cstate: current_term: 2 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:24.856465 13222 raft_consensus.cc:2270] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:24.857401 13222 ts_tablet_manager.cc:1905] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:24.868525 8063 consensus_queue.cc:237] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } }
I20250627 13:54:24.868753 13222 ts_tablet_manager.cc:1918] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20250627 13:54:24.868873 13222 log.cc:1199] T d2eb537d49804e1b8ec5a217b1564988 P 956ff058e6d64579a1963aabdb6d466b: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal/wals/d2eb537d49804e1b8ec5a217b1564988
I20250627 13:54:24.869627 7645 catalog_manager.cc:4928] TS 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): tablet d2eb537d49804e1b8ec5a217b1564988 (table rebalance_test_table_0 [id=d8bfc9268a4a4c2c92bc20d56ac73086]) successfully deleted
I20250627 13:54:24.871234 7931 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250627 13:54:24.871779 9972 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250627 13:54:24.872537 9839 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250627 13:54:24.874307 12848 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250627 13:54:24.875212 13188 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250627 13:54:24.876690 12180 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250627 13:54:24.882645 7911 tablet_service.cc:1515] Processing DeleteTablet for tablet 03640a5124d2405faf4a26d0e22e1ffc with delete_type TABLET_DATA_TOMBSTONED (TS a41a89ae8ed743edb2b5cf4490d8c854 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:59302
I20250627 13:54:24.890043 13188 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:24.890959 7931 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:24.896845 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 118d5145b23f43798e1c727f2a9e9f92 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:24.898675 9839 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:24.899664 7653 catalog_manager.cc:5582] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 12 to 13, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:24.909704 8171 tablet_service.cc:1515] Processing DeleteTablet for tablet 118d5145b23f43798e1c727f2a9e9f92 with delete_type TABLET_DATA_TOMBSTONED (TS e4c831df4da841929d13d6c783099b0e not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:48616
I20250627 13:54:24.913292 13084 raft_consensus.cc:1062] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e: attempting to promote NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a to VOTER
I20250627 13:54:24.914304 13084 consensus_queue.cc:237] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:24.914458 8063 consensus_queue.cc:237] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } }
I20250627 13:54:24.916894 7931 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250627 13:54:24.917606 13190 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250627 13:54:24.917932 9973 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:24.919554 9839 raft_consensus.cc:1273] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250627 13:54:24.920230 12180 consensus_queue.cc:1035] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250627 13:54:24.920892 12180 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:24.926378 7927 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:24.927040 9839 raft_consensus.cc:2953] T 118d5145b23f43798e1c727f2a9e9f92 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:24.928576 13232 tablet_replica.cc:331] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e: stopping tablet replica
I20250627 13:54:24.929916 13232 raft_consensus.cc:2241] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:24.930774 13232 pending_rounds.cc:70] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e: Trying to abort 1 pending ops.
I20250627 13:54:24.931464 13232 pending_rounds.cc:77] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e: Aborting op as it isn't in flight: id { term: 1 index: 12 } timestamp: 7172228975644950528 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "118d5145b23f43798e1c727f2a9e9f92" old_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } } new_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } } }
I20250627 13:54:24.932947 13232 raft_consensus.cc:2915] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Aborting config change with OpId 1.12: Aborted: Op aborted
I20250627 13:54:24.933472 13232 raft_consensus.cc:2887] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250627 13:54:24.933912 13232 raft_consensus.cc:2270] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:24.935858 7656 catalog_manager.cc:5582] T 118d5145b23f43798e1c727f2a9e9f92 P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index 13 to 14, VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:24.947280 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 118d5145b23f43798e1c727f2a9e9f92 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:24.948189 13232 ts_tablet_manager.cc:1905] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:24.956991 13229 tablet_replica.cc:331] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854: stopping tablet replica
I20250627 13:54:24.957402 13229 raft_consensus.cc:2241] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:24.957695 13229 raft_consensus.cc:2270] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:24.958842 13229 ts_tablet_manager.cc:1905] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:24.976655 13229 ts_tablet_manager.cc:1918] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20250627 13:54:24.977061 13229 log.cc:1199] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/wals/03640a5124d2405faf4a26d0e22e1ffc
I20250627 13:54:24.977813 7647 catalog_manager.cc:4928] TS a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675): tablet 03640a5124d2405faf4a26d0e22e1ffc (table rebalance_test_table_1 [id=8961aa0df2f1488f82cd55cfa22ad214]) successfully deleted
I20250627 13:54:25.017089 13232 ts_tablet_manager.cc:1918] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20250627 13:54:25.017519 9709 raft_consensus.cc:1273] T da90822ff15e4e5da05114ef38d3970a P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:25.024348 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:25.024905 9954 tablet_service.cc:1515] Processing DeleteTablet for tablet 118d5145b23f43798e1c727f2a9e9f92 with delete_type TABLET_DATA_TOMBSTONED (TS 6686c6c365bf4f4582058890161676fe not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:35810
I20250627 13:54:25.029796 9953 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:25.018304 13088 consensus_queue.cc:1035] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:25.033805 13232 log.cc:1199] T 118d5145b23f43798e1c727f2a9e9f92 P e4c831df4da841929d13d6c783099b0e: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal/wals/118d5145b23f43798e1c727f2a9e9f92
I20250627 13:54:25.035610 7648 catalog_manager.cc:4928] TS e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): tablet 118d5145b23f43798e1c727f2a9e9f92 (table rebalance_test_table_5 [id=dd79cdfc35164ae6a649d3a77ee397c2]) successfully deleted
I20250627 13:54:25.036108 9839 raft_consensus.cc:1273] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:25.036190 7927 raft_consensus.cc:1273] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:25.036476 13251 tablet_replica.cc:331] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe: stopping tablet replica
I20250627 13:54:25.036747 13083 consensus_queue.cc:1035] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:25.037144 13251 raft_consensus.cc:2241] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:25.037313 12836 consensus_queue.cc:1035] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:25.037869 13251 raft_consensus.cc:2270] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:25.040199 12941 raft_consensus.cc:1062] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096: attempting to promote NON_VOTER 2a1282ebe92a4c9a92eed3b13f69012a to VOTER
I20250627 13:54:25.041384 12941 consensus_queue.cc:237] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:25.041603 13251 ts_tablet_manager.cc:1905] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:25.053233 13251 ts_tablet_manager.cc:1918] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.13
I20250627 13:54:25.053488 13251 log.cc:1199] T 118d5145b23f43798e1c727f2a9e9f92 P 6686c6c365bf4f4582058890161676fe: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/wal/wals/118d5145b23f43798e1c727f2a9e9f92
I20250627 13:54:25.056222 7648 catalog_manager.cc:4928] TS 6686c6c365bf4f4582058890161676fe (127.0.103.71:37859): tablet 118d5145b23f43798e1c727f2a9e9f92 (table rebalance_test_table_5 [id=dd79cdfc35164ae6a649d3a77ee397c2]) successfully deleted
I20250627 13:54:25.057767 9973 raft_consensus.cc:1273] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:25.058358 12942 consensus_queue.cc:1035] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:25.060320 9711 raft_consensus.cc:1273] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 LEARNER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:25.060784 12942 consensus_queue.cc:1035] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:25.068351 13083 raft_consensus.cc:2953] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:25.075055 9709 raft_consensus.cc:2953] T da90822ff15e4e5da05114ef38d3970a P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:25.076658 7931 raft_consensus.cc:2953] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:25.078343 13188 raft_consensus.cc:1062] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER 83574c1bf8ea45adaff939158f93c096 to VOTER
I20250627 13:54:25.079236 13188 consensus_queue.cc:237] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } }
I20250627 13:54:25.080919 7656 catalog_manager.cc:5582] T da90822ff15e4e5da05114ef38d3970a P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: config changed from index 9 to 10, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:25.083737 9839 raft_consensus.cc:2953] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:25.086017 7927 raft_consensus.cc:1273] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250627 13:54:25.095268 12942 consensus_queue.cc:1035] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:25.096104 12942 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:25.098518 7800 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250627 13:54:25.099128 12180 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:25.100620 9972 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:25.107296 8190 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250627 13:54:25.107769 9839 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250627 13:54:25.107674 7653 catalog_manager.cc:5582] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe reported cstate change: config changed from index 7 to 8, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:25.107923 12180 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:25.109685 12848 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:25.115247 13188 raft_consensus.cc:1025] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: attempt to promote peer e4c831df4da841929d13d6c783099b0e: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250627 13:54:25.120645 10104 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250627 13:54:25.124763 13188 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:25.138269 9709 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:25.148684 13201 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:25.149847 9841 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:25.155334 8190 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 LEARNER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:25.157469 7653 catalog_manager.cc:5582] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b reported cstate change: config changed from index 11 to 12, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:25.161548 13188 raft_consensus.cc:1062] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER e4c831df4da841929d13d6c783099b0e to VOTER
I20250627 13:54:25.162518 13188 consensus_queue.cc:237] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } }
I20250627 13:54:25.165890 9841 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250627 13:54:25.166536 13201 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250627 13:54:25.167228 8190 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250627 13:54:25.167557 10104 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:25.170404 12848 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250627 13:54:25.177211 9822 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:25.190409 10104 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250627 13:54:25.193779 13201 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:25.194407 8190 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
W20250627 13:54:25.195416 7738 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.69:33101, user_credentials={real_user=slave}} blocked reactor thread for 51352.4us
I20250627 13:54:25.199270 9839 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:25.199772 13190 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250627 13:54:25.200866 7659 catalog_manager.cc:5582] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 12 to 13, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:25.205766 10104 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:25.223512 7931 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:25.232882 7801 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250627 13:54:25.234504 7782 tablet_service.cc:1430] Tablet server has 8 leaders and 0 scanners
I20250627 13:54:25.235056 13190 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:25.236384 7801 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:25.241658 7801 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:25.265887 8041 tablet_service.cc:1430] Tablet server has 10 leaders and 0 scanners
I20250627 13:54:25.334774 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:25.373692 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:25.401628 10083 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:25.437304 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:25.453776 8041 tablet_service.cc:1430] Tablet server has 10 leaders and 0 scanners
I20250627 13:54:25.454226 9691 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:25.467585 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:25.474130 9822 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:25.474982 7782 tablet_service.cc:1430] Tablet server has 8 leaders and 0 scanners
I20250627 13:54:25.532545 8171 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:25.538059 10083 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:25.647291 8063 consensus_queue.cc:237] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } }
I20250627 13:54:25.650763 8190 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250627 13:54:25.651449 13201 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250627 13:54:25.659548 7800 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250627 13:54:25.662066 13201 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250627 13:54:25.654062 10102 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250627 13:54:25.666831 13188 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250627 13:54:25.674604 13201 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:25.678421 7800 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:25.681716 10104 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:25.690805 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 644985e3086d4614b6598a727cb60c1d with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:25.692027 8190 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:25.702589 7659 catalog_manager.cc:5582] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 13 to 14, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:25.762308 9822 tablet_service.cc:1515] Processing DeleteTablet for tablet 644985e3086d4614b6598a727cb60c1d with delete_type TABLET_DATA_TOMBSTONED (TS 956ff058e6d64579a1963aabdb6d466b not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:55228
I20250627 13:54:25.768460 13351 tablet_replica.cc:331] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: stopping tablet replica
I20250627 13:54:25.768752 13351 raft_consensus.cc:2241] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:25.769260 13351 raft_consensus.cc:2270] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:25.773778 13351 ts_tablet_manager.cc:1905] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:25.780479 8062 consensus_queue.cc:237] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } }
I20250627 13:54:25.783691 7800 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250627 13:54:25.784204 13188 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250627 13:54:25.790479 8189 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250627 13:54:25.790973 13351 ts_tablet_manager.cc:1918] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.13
I20250627 13:54:25.791124 13351 log.cc:1199] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal/wals/644985e3086d4614b6598a727cb60c1d
I20250627 13:54:25.791657 13190 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250627 13:54:25.795234 13190 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:25.798544 8189 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:25.802635 7645 catalog_manager.cc:4928] TS 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): tablet 644985e3086d4614b6598a727cb60c1d (table rebalance_test_table_1 [id=8961aa0df2f1488f82cd55cfa22ad214]) successfully deleted
I20250627 13:54:25.805768 7800 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:25.810062 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 644985e3086d4614b6598a727cb60c1d with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:25.821636 7656 catalog_manager.cc:5582] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 14 to 15, VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:25.827916 10083 tablet_service.cc:1515] Processing DeleteTablet for tablet 644985e3086d4614b6598a727cb60c1d with delete_type TABLET_DATA_TOMBSTONED (TS dede81a8c0164427a61532d01b6fc157 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:56490
I20250627 13:54:25.829576 7658 catalog_manager.cc:5129] ChangeConfig:REMOVE_PEER RPC for tablet 644985e3086d4614b6598a727cb60c1d with cas_config_opid_index 14: aborting the task: latest config opid_index 15; task opid_index 14
I20250627 13:54:25.831390 13360 tablet_replica.cc:331] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157: stopping tablet replica
I20250627 13:54:25.831619 13360 raft_consensus.cc:2241] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:25.832033 13360 raft_consensus.cc:2270] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:25.834685 13360 ts_tablet_manager.cc:1905] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:25.843991 13360 ts_tablet_manager.cc:1918] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.14
I20250627 13:54:25.844166 13360 log.cc:1199] T 644985e3086d4614b6598a727cb60c1d P dede81a8c0164427a61532d01b6fc157: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/wal/wals/644985e3086d4614b6598a727cb60c1d
I20250627 13:54:25.845032 7647 catalog_manager.cc:4928] TS dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213): tablet 644985e3086d4614b6598a727cb60c1d (table rebalance_test_table_1 [id=8961aa0df2f1488f82cd55cfa22ad214]) successfully deleted
I20250627 13:54:25.857286 8189 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "e3053b529a264e8697a1df9e17eeea81"
dest_uuid: "e4c831df4da841929d13d6c783099b0e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60588
I20250627 13:54:25.857530 8189 raft_consensus.cc:604] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Received request to transfer leadership
I20250627 13:54:25.921000 10102 consensus_queue.cc:237] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } }
W20250627 13:54:25.932674 10040 consensus_peers.cc:487] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 -> Peer e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): Couldn't send request to peer e4c831df4da841929d13d6c783099b0e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:25.935005 8062 raft_consensus.cc:1273] T 5f176b66fdce4398be3492a5f3b4ea27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Refusing update from remote peer dede81a8c0164427a61532d01b6fc157: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20250627 13:54:25.935638 7800 raft_consensus.cc:1273] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Refusing update from remote peer dede81a8c0164427a61532d01b6fc157: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20250627 13:54:25.936992 12844 consensus_queue.cc:1035] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:25.937575 12844 consensus_queue.cc:1035] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250627 13:54:25.947901 12844 raft_consensus.cc:2953] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
I20250627 13:54:25.949713 8062 raft_consensus.cc:2953] T 5f176b66fdce4398be3492a5f3b4ea27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
I20250627 13:54:25.956459 7658 catalog_manager.cc:5582] T 5f176b66fdce4398be3492a5f3b4ea27 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 7 to 8, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "dede81a8c0164427a61532d01b6fc157" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
I20250627 13:54:25.968556 7800 raft_consensus.cc:2953] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
I20250627 13:54:26.044162 7927 consensus_queue.cc:237] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:26.048820 9711 raft_consensus.cc:1273] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:26.049337 9839 raft_consensus.cc:1273] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:26.049400 8062 raft_consensus.cc:1273] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250627 13:54:26.049849 12115 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:26.050115 13375 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:26.050472 13377 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:26.060812 13375 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:26.069068 8063 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "83cfc975906745a09c5e1dcfac812ae2"
dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34808
I20250627 13:54:26.069283 8063 raft_consensus.cc:604] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Received request to transfer leadership
I20250627 13:54:26.071158 8062 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:26.074309 9839 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:26.081225 7931 consensus_queue.cc:237] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:26.081959 9711 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:26.092839 7658 catalog_manager.cc:5582] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index 8 to 9. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:26.098857 7927 consensus_queue.cc:237] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:26.105020 9708 raft_consensus.cc:1273] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:26.105666 12115 consensus_queue.cc:1035] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:26.106024 7800 raft_consensus.cc:1273] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20250627 13:54:26.108024 7866 consensus_peers.cc:487] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b265cfda4bfe48908659208ec8eeab2d. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:26.108383 12115 consensus_queue.cc:1035] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:26.114764 13377 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:26.118364 9709 raft_consensus.cc:1273] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:26.123343 8063 raft_consensus.cc:1273] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:26.127696 13444 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:26.128436 12115 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:26.128557 7800 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:26.130630 12115 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:26.135356 8063 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:26.136726 9709 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:26.140540 7647 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet cff879c8e57d42c28440b3bd310f1ba9 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:26.141407 7655 catalog_manager.cc:5582] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 9 to 10, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:26.146662 7659 catalog_manager.cc:5582] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: config changed from index 9 to 10, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:26.150718 9822 tablet_service.cc:1515] Processing DeleteTablet for tablet cff879c8e57d42c28440b3bd310f1ba9 with delete_type TABLET_DATA_TOMBSTONED (TS 956ff058e6d64579a1963aabdb6d466b not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55228
I20250627 13:54:26.151069 13351 tablet_replica.cc:331] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b: stopping tablet replica
I20250627 13:54:26.151331 13351 raft_consensus.cc:2241] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:26.151643 13351 raft_consensus.cc:2270] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:26.152590 13351 ts_tablet_manager.cc:1905] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:26.153753 7927 consensus_queue.cc:237] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
W20250627 13:54:26.156740 7868 consensus_peers.cc:487] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 -> Peer dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213): Couldn't send request to peer dede81a8c0164427a61532d01b6fc157. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cff879c8e57d42c28440b3bd310f1ba9. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:26.157114 8063 raft_consensus.cc:1273] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:26.157874 13375 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:26.162066 13351 ts_tablet_manager.cc:1918] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20250627 13:54:26.167062 13377 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:26.170286 8062 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:26.171576 13351 log.cc:1199] T cff879c8e57d42c28440b3bd310f1ba9 P 956ff058e6d64579a1963aabdb6d466b: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal/wals/cff879c8e57d42c28440b3bd310f1ba9
I20250627 13:54:26.181594 7656 catalog_manager.cc:5582] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 10 to 11, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:26.181739 9709 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:26.188248 13190 raft_consensus.cc:991] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660: : Instructing follower a41a89ae8ed743edb2b5cf4490d8c854 to start an election
I20250627 13:54:26.188405 13190 raft_consensus.cc:1079] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Signalling peer a41a89ae8ed743edb2b5cf4490d8c854 to start an election
I20250627 13:54:26.189096 7647 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet cff879c8e57d42c28440b3bd310f1ba9 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250627 13:54:26.190325 7927 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "83cfc975906745a09c5e1dcfac812ae2"
dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
from {username='slave'} at 127.0.103.67:51721
I20250627 13:54:26.190552 7927 raft_consensus.cc:491] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:54:26.190708 7927 raft_consensus.cc:3058] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:26.195730 7927 raft_consensus.cc:513] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:26.196905 8062 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "83cfc975906745a09c5e1dcfac812ae2" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660"
I20250627 13:54:26.197104 8062 raft_consensus.cc:3053] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Stepping down as leader of term 1
I20250627 13:54:26.197191 8062 raft_consensus.cc:738] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Running, Role: LEADER
I20250627 13:54:26.197119 10102 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "83cfc975906745a09c5e1dcfac812ae2" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "dede81a8c0164427a61532d01b6fc157"
I20250627 13:54:26.197345 10102 raft_consensus.cc:3058] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:26.197482 8062 consensus_queue.cc:260] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:26.197907 8062 raft_consensus.cc:3058] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:26.200887 10102 raft_consensus.cc:2466] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41a89ae8ed743edb2b5cf4490d8c854 in term 2.
I20250627 13:54:26.201308 8062 raft_consensus.cc:2466] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41a89ae8ed743edb2b5cf4490d8c854 in term 2.
I20250627 13:54:26.201779 7927 leader_election.cc:290] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 2 election: Requested vote from peers 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933), dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213), 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647)
I20250627 13:54:26.201800 7866 leader_election.cc:304] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 9cf479bdcd0242738d2a77db4ad9f660, a41a89ae8ed743edb2b5cf4490d8c854, dede81a8c0164427a61532d01b6fc157; no voters:
I20250627 13:54:26.202286 13375 raft_consensus.cc:2802] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:54:26.202454 13375 raft_consensus.cc:695] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 LEADER]: Becoming Leader. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Running, Role: LEADER
I20250627 13:54:26.202589 9839 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "83cfc975906745a09c5e1dcfac812ae2" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "956ff058e6d64579a1963aabdb6d466b"
I20250627 13:54:26.202775 9839 raft_consensus.cc:3058] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:26.202720 13375 consensus_queue.cc:237] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:26.204200 9708 raft_consensus.cc:1273] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:26.204787 13377 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:26.206295 9839 raft_consensus.cc:2466] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41a89ae8ed743edb2b5cf4490d8c854 in term 2.
I20250627 13:54:26.214879 7645 catalog_manager.cc:4928] TS 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): tablet cff879c8e57d42c28440b3bd310f1ba9 (table rebalance_test_table_4 [id=4d92d465970b4ec4b5eace263b25362a]) successfully deleted
I20250627 13:54:26.216447 9708 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } }
I20250627 13:54:26.222085 7927 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "cff879c8e57d42c28440b3bd310f1ba9"
dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35426
I20250627 13:54:26.222277 7927 raft_consensus.cc:604] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Received request to transfer leadership
I20250627 13:54:26.225929 7655 catalog_manager.cc:5582] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: term changed from 1 to 2, leader changed from 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) to a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66). New cstate: current_term: 2 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:26.226471 13375 raft_consensus.cc:991] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854: : Instructing follower 2a1282ebe92a4c9a92eed3b13f69012a to start an election
I20250627 13:54:26.226620 13444 raft_consensus.cc:1079] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Signalling peer 2a1282ebe92a4c9a92eed3b13f69012a to start an election
I20250627 13:54:26.227389 9709 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "cff879c8e57d42c28440b3bd310f1ba9"
dest_uuid: "2a1282ebe92a4c9a92eed3b13f69012a"
from {username='slave'} at 127.0.103.66:51153
I20250627 13:54:26.227682 9709 raft_consensus.cc:491] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:54:26.227825 9709 raft_consensus.cc:3058] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:26.231169 9709 raft_consensus.cc:513] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:26.232960 9709 leader_election.cc:290] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [CANDIDATE]: Term 2 election: Requested vote from peers a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675), 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:26.235206 7927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cff879c8e57d42c28440b3bd310f1ba9" candidate_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
I20250627 13:54:26.235412 7927 raft_consensus.cc:3053] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Stepping down as leader of term 1
I20250627 13:54:26.235486 7927 raft_consensus.cc:738] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Running, Role: LEADER
I20250627 13:54:26.235827 7927 consensus_queue.cc:260] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:26.236287 7927 raft_consensus.cc:3058] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:26.239573 7927 raft_consensus.cc:2466] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1282ebe92a4c9a92eed3b13f69012a in term 2.
I20250627 13:54:26.240213 9646 leader_election.cc:304] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a1282ebe92a4c9a92eed3b13f69012a, a41a89ae8ed743edb2b5cf4490d8c854; no voters:
I20250627 13:54:26.240634 13434 raft_consensus.cc:2802] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:54:26.241835 8062 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cff879c8e57d42c28440b3bd310f1ba9" candidate_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660"
I20250627 13:54:26.242069 8062 raft_consensus.cc:3058] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:26.242607 13434 raft_consensus.cc:695] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 LEADER]: Becoming Leader. State: Replica: 2a1282ebe92a4c9a92eed3b13f69012a, State: Running, Role: LEADER
I20250627 13:54:26.243120 13434 consensus_queue.cc:237] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:26.247956 7658 catalog_manager.cc:5582] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: term changed from 1 to 2, leader changed from a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) to 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69). New cstate: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:26.250276 8062 raft_consensus.cc:2466] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1282ebe92a4c9a92eed3b13f69012a in term 2.
W20250627 13:54:26.264220 8129 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.69:33101, user_credentials={real_user=slave}} blocked reactor thread for 75104.9us
I20250627 13:54:26.277916 13445 raft_consensus.cc:991] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: : Instructing follower 2a1282ebe92a4c9a92eed3b13f69012a to start an election
I20250627 13:54:26.277819 8062 consensus_queue.cc:237] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } }
I20250627 13:54:26.278136 13445 raft_consensus.cc:1079] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Signalling peer 2a1282ebe92a4c9a92eed3b13f69012a to start an election
I20250627 13:54:26.281795 9708 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "e3053b529a264e8697a1df9e17eeea81"
dest_uuid: "2a1282ebe92a4c9a92eed3b13f69012a"
from {username='slave'} at 127.0.103.68:51963
I20250627 13:54:26.282020 9708 raft_consensus.cc:491] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:54:26.282146 9708 raft_consensus.cc:3058] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:26.285049 9708 raft_consensus.cc:513] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } }
I20250627 13:54:26.286463 8189 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3053b529a264e8697a1df9e17eeea81" candidate_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: true dest_uuid: "e4c831df4da841929d13d6c783099b0e"
I20250627 13:54:26.286669 8189 raft_consensus.cc:3053] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Stepping down as leader of term 1
I20250627 13:54:26.286751 8189 raft_consensus.cc:738] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Running, Role: LEADER
I20250627 13:54:26.287030 8189 consensus_queue.cc:260] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } }
I20250627 13:54:26.287354 8189 raft_consensus.cc:3058] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Advancing to term 2
W20250627 13:54:26.287408 7999 consensus_peers.cc:487] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): Couldn't send request to peer e4c831df4da841929d13d6c783099b0e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:26.288542 9973 raft_consensus.cc:1273] T 9196497cb11f4ccbbd62695253d4c8c9 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:26.289096 13201 consensus_queue.cc:1035] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:26.290489 8189 raft_consensus.cc:2466] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1282ebe92a4c9a92eed3b13f69012a in term 2.
I20250627 13:54:26.292762 9839 raft_consensus.cc:1273] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:26.293468 13201 consensus_queue.cc:1035] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:26.294891 9708 leader_election.cc:290] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [CANDIDATE]: Term 2 election: Requested vote from peers e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), 6686c6c365bf4f4582058890161676fe (127.0.103.71:37859), a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:54:26.295397 7927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3053b529a264e8697a1df9e17eeea81" candidate_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: true dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
I20250627 13:54:26.295604 7927 raft_consensus.cc:3058] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:26.299615 7927 raft_consensus.cc:2466] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1282ebe92a4c9a92eed3b13f69012a in term 2.
I20250627 13:54:26.300084 9646 leader_election.cc:304] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2a1282ebe92a4c9a92eed3b13f69012a, a41a89ae8ed743edb2b5cf4490d8c854, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:54:26.300525 13434 raft_consensus.cc:2802] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:54:26.300704 13434 raft_consensus.cc:695] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 LEADER]: Becoming Leader. State: Replica: 2a1282ebe92a4c9a92eed3b13f69012a, State: Running, Role: LEADER
I20250627 13:54:26.301085 13434 consensus_queue.cc:237] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } }
I20250627 13:54:26.325289 13201 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
I20250627 13:54:26.331036 7658 catalog_manager.cc:5582] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: term changed from 1 to 2, leader changed from e4c831df4da841929d13d6c783099b0e (127.0.103.68) to 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69). New cstate: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:26.336007 9972 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
I20250627 13:54:26.344810 7658 catalog_manager.cc:5582] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 9 to 10, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:26.373517 9841 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
I20250627 13:54:26.413424 9972 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3053b529a264e8697a1df9e17eeea81" candidate_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: true dest_uuid: "6686c6c365bf4f4582058890161676fe"
I20250627 13:54:26.413836 9972 raft_consensus.cc:3058] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:26.417857 9972 raft_consensus.cc:2466] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1282ebe92a4c9a92eed3b13f69012a in term 2.
I20250627 13:54:26.472752 13511 ts_tablet_manager.cc:927] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e: Initiating tablet copy from peer dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213)
I20250627 13:54:26.473662 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:26.486553 13511 tablet_copy_client.cc:323] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.72:35213
I20250627 13:54:26.488322 10124 tablet_copy_service.cc:140] P dede81a8c0164427a61532d01b6fc157: Received BeginTabletCopySession request for tablet 5f176b66fdce4398be3492a5f3b4ea27 from peer e4c831df4da841929d13d6c783099b0e ({username='slave'} at 127.0.103.68:39671)
I20250627 13:54:26.488550 10124 tablet_copy_service.cc:161] P dede81a8c0164427a61532d01b6fc157: Beginning new tablet copy session on tablet 5f176b66fdce4398be3492a5f3b4ea27 from peer e4c831df4da841929d13d6c783099b0e at {username='slave'} at 127.0.103.68:39671: session id = e4c831df4da841929d13d6c783099b0e-5f176b66fdce4398be3492a5f3b4ea27
I20250627 13:54:26.490619 7782 tablet_service.cc:1430] Tablet server has 8 leaders and 0 scanners
I20250627 13:54:26.493363 10124 tablet_copy_source_session.cc:215] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:26.503073 7800 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "40431e3bd7244904a4e65f59477bd21d"
dest_uuid: "83574c1bf8ea45adaff939158f93c096"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58880
I20250627 13:54:26.503285 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:26.503440 7800 raft_consensus.cc:604] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Received request to transfer leadership
I20250627 13:54:26.507014 13511 ts_tablet_manager.cc:1905] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250627 13:54:26.539285 13265 raft_consensus.cc:991] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096: : Instructing follower 2a1282ebe92a4c9a92eed3b13f69012a to start an election
I20250627 13:54:26.539453 13265 raft_consensus.cc:1079] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Signalling peer 2a1282ebe92a4c9a92eed3b13f69012a to start an election
I20250627 13:54:26.540491 9708 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "40431e3bd7244904a4e65f59477bd21d"
dest_uuid: "2a1282ebe92a4c9a92eed3b13f69012a"
from {username='slave'} at 127.0.103.65:56847
I20250627 13:54:26.540731 9708 raft_consensus.cc:491] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:54:26.540864 9708 raft_consensus.cc:3058] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:26.541041 13511 ts_tablet_manager.cc:1918] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20250627 13:54:26.541442 13511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f176b66fdce4398be3492a5f3b4ea27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:26.544474 9708 raft_consensus.cc:513] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:26.547426 9708 leader_election.cc:290] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [CANDIDATE]: Term 2 election: Requested vote from peers 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467), a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675), 6686c6c365bf4f4582058890161676fe (127.0.103.71:37859)
I20250627 13:54:26.547801 7927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40431e3bd7244904a4e65f59477bd21d" candidate_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
I20250627 13:54:26.548023 7927 raft_consensus.cc:3058] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:26.548259 7800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40431e3bd7244904a4e65f59477bd21d" candidate_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "83574c1bf8ea45adaff939158f93c096"
I20250627 13:54:26.548260 7801 consensus_queue.cc:237] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: NON_VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: true } }
I20250627 13:54:26.548550 7800 raft_consensus.cc:3053] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Stepping down as leader of term 1
I20250627 13:54:26.548636 7800 raft_consensus.cc:738] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Running, Role: LEADER
I20250627 13:54:26.548985 7800 consensus_queue.cc:260] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:26.549407 7800 raft_consensus.cc:3058] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:26.551602 7927 raft_consensus.cc:2466] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1282ebe92a4c9a92eed3b13f69012a in term 2.
I20250627 13:54:26.553315 7800 raft_consensus.cc:2466] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1282ebe92a4c9a92eed3b13f69012a in term 2.
I20250627 13:54:26.553815 9644 leader_election.cc:304] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2a1282ebe92a4c9a92eed3b13f69012a, 83574c1bf8ea45adaff939158f93c096, a41a89ae8ed743edb2b5cf4490d8c854; no voters:
I20250627 13:54:26.554160 13434 raft_consensus.cc:2802] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:54:26.554323 13434 raft_consensus.cc:695] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 LEADER]: Becoming Leader. State: Replica: 2a1282ebe92a4c9a92eed3b13f69012a, State: Running, Role: LEADER
I20250627 13:54:26.555214 13434 consensus_queue.cc:237] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:26.557135 9972 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40431e3bd7244904a4e65f59477bd21d" candidate_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "6686c6c365bf4f4582058890161676fe"
I20250627 13:54:26.557384 9972 raft_consensus.cc:3058] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:26.560452 13511 tablet_copy_client.cc:806] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:26.560750 13511 tablet_copy_client.cc:670] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:26.560853 9972 raft_consensus.cc:2466] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1282ebe92a4c9a92eed3b13f69012a in term 2.
I20250627 13:54:26.563823 13511 tablet_copy_client.cc:538] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20250627 13:54:26.566330 7735 consensus_peers.cc:487] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 -> Peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933): Couldn't send request to peer 9cf479bdcd0242738d2a77db4ad9f660. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:26.567817 7927 raft_consensus.cc:1273] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:26.568408 12942 consensus_queue.cc:1035] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:26.570626 8041 tablet_service.cc:1430] Tablet server has 9 leaders and 0 scanners
I20250627 13:54:26.581387 7658 catalog_manager.cc:5582] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: term changed from 1 to 2, leader changed from 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) to 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69). New cstate: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:26.587651 9691 tablet_service.cc:1430] Tablet server has 3 leaders and 0 scanners
I20250627 13:54:26.608470 9839 raft_consensus.cc:1273] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:26.609836 12942 consensus_queue.cc:1035] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:26.618880 13528 ts_tablet_manager.cc:927] T b265cfda4bfe48908659208ec8eeab2d P 956ff058e6d64579a1963aabdb6d466b: Initiating tablet copy from peer a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:54:26.618988 12942 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: NON_VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: true } } }
I20250627 13:54:26.628309 7658 catalog_manager.cc:5582] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 9 to 10, NON_VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) added. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: NON_VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:26.632169 8171 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:26.640375 9841 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: NON_VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: true } } }
I20250627 13:54:26.648663 7927 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: NON_VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: true } } }
I20250627 13:54:26.653987 8062 raft_consensus.cc:1273] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250627 13:54:26.654656 13375 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:26.657428 7933 raft_consensus.cc:1273] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250627 13:54:26.658469 13434 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:26.676615 13528 tablet_copy_client.cc:323] T b265cfda4bfe48908659208ec8eeab2d P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:43675
I20250627 13:54:26.677588 7953 tablet_copy_service.cc:140] P a41a89ae8ed743edb2b5cf4490d8c854: Received BeginTabletCopySession request for tablet b265cfda4bfe48908659208ec8eeab2d from peer 956ff058e6d64579a1963aabdb6d466b ({username='slave'} at 127.0.103.70:60171)
I20250627 13:54:26.677839 7953 tablet_copy_service.cc:161] P a41a89ae8ed743edb2b5cf4490d8c854: Beginning new tablet copy session on tablet b265cfda4bfe48908659208ec8eeab2d from peer 956ff058e6d64579a1963aabdb6d466b at {username='slave'} at 127.0.103.70:60171: session id = 956ff058e6d64579a1963aabdb6d466b-b265cfda4bfe48908659208ec8eeab2d
I20250627 13:54:26.682229 8062 raft_consensus.cc:1273] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250627 13:54:26.682677 7953 tablet_copy_source_session.cc:215] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:26.682850 13434 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:26.688035 13528 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b265cfda4bfe48908659208ec8eeab2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:26.703708 10083 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:26.718703 8189 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 2 index: 17. (index mismatch)
I20250627 13:54:26.719457 13461 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250627 13:54:26.721704 9839 raft_consensus.cc:1273] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250627 13:54:26.722481 13375 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:26.737231 10102 raft_consensus.cc:1273] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250627 13:54:26.737887 13375 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:26.755383 13547 ts_tablet_manager.cc:927] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:26.777988 13511 tablet_bootstrap.cc:492] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
W20250627 13:54:26.778173 9646 consensus_peers.cc:487] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a -> Peer dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213): Couldn't send request to peer dede81a8c0164427a61532d01b6fc157. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cff879c8e57d42c28440b3bd310f1ba9. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:26.788794 13547 tablet_copy_client.cc:323] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:26.789517 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet 9196497cb11f4ccbbd62695253d4c8c9 from peer e4c831df4da841929d13d6c783099b0e ({username='slave'} at 127.0.103.68:59937)
I20250627 13:54:26.789731 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet 9196497cb11f4ccbbd62695253d4c8c9 from peer e4c831df4da841929d13d6c783099b0e at {username='slave'} at 127.0.103.68:59937: session id = e4c831df4da841929d13d6c783099b0e-9196497cb11f4ccbbd62695253d4c8c9
I20250627 13:54:26.792826 8082 tablet_copy_source_session.cc:215] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:26.793931 13547 ts_tablet_manager.cc:1905] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250627 13:54:26.800014 13547 ts_tablet_manager.cc:1918] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20250627 13:54:26.800285 13547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9196497cb11f4ccbbd62695253d4c8c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:26.802963 13528 tablet_copy_client.cc:806] T b265cfda4bfe48908659208ec8eeab2d P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:26.803299 13528 tablet_copy_client.cc:670] T b265cfda4bfe48908659208ec8eeab2d P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:26.806972 13528 tablet_copy_client.cc:538] T b265cfda4bfe48908659208ec8eeab2d P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:26.808099 9822 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:26.815243 13528 tablet_bootstrap.cc:492] T b265cfda4bfe48908659208ec8eeab2d P 956ff058e6d64579a1963aabdb6d466b: Bootstrap starting.
I20250627 13:54:26.815461 13547 tablet_copy_client.cc:806] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:26.815793 13547 tablet_copy_client.cc:670] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:26.828902 7933 consensus_queue.cc:237] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:26.832770 10102 raft_consensus.cc:1273] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250627 13:54:26.833534 13546 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:26.836406 13546 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:26.838866 9839 raft_consensus.cc:1273] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250627 13:54:26.839628 13375 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:26.840493 10102 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:26.841110 9841 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:26.847019 7656 catalog_manager.cc:5582] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157 reported cstate change: config changed from index 8 to 10, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New cstate: current_term: 2 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:26.848536 7647 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 83cfc975906745a09c5e1dcfac812ae2 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:26.852344 8041 tablet_service.cc:1515] Processing DeleteTablet for tablet 83cfc975906745a09c5e1dcfac812ae2 with delete_type TABLET_DATA_TOMBSTONED (TS 9cf479bdcd0242738d2a77db4ad9f660 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:59760
I20250627 13:54:26.856518 9972 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 2 index: 17. (index mismatch)
I20250627 13:54:26.857270 13461 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250627 13:54:26.874645 13566 tablet_replica.cc:331] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660: stopping tablet replica
I20250627 13:54:26.874857 13547 tablet_copy_client.cc:538] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:26.875119 13566 raft_consensus.cc:2241] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:26.875365 13566 raft_consensus.cc:2270] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:26.882687 13547 tablet_bootstrap.cc:492] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
I20250627 13:54:26.888384 13566 ts_tablet_manager.cc:1905] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:26.911000 7933 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 2 index: 17. (index mismatch)
I20250627 13:54:26.911666 13461 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250627 13:54:26.921006 13566 ts_tablet_manager.cc:1918] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20250627 13:54:26.921231 13566 log.cc:1199] T 83cfc975906745a09c5e1dcfac812ae2 P 9cf479bdcd0242738d2a77db4ad9f660: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal/wals/83cfc975906745a09c5e1dcfac812ae2
I20250627 13:54:26.922287 7645 catalog_manager.cc:4928] TS 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933): tablet 83cfc975906745a09c5e1dcfac812ae2 (table rebalance_test_table_9 [id=19e5686b50fe4187a733aa6413ef2ce0]) successfully deleted
I20250627 13:54:26.970618 13528 tablet_bootstrap.cc:492] T b265cfda4bfe48908659208ec8eeab2d P 956ff058e6d64579a1963aabdb6d466b: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:26.979566 13528 tablet_bootstrap.cc:492] T b265cfda4bfe48908659208ec8eeab2d P 956ff058e6d64579a1963aabdb6d466b: Bootstrap complete.
I20250627 13:54:26.980885 13528 ts_tablet_manager.cc:1397] T b265cfda4bfe48908659208ec8eeab2d P 956ff058e6d64579a1963aabdb6d466b: Time spent bootstrapping tablet: real 0.166s user 0.063s sys 0.004s
I20250627 13:54:26.980990 9708 consensus_queue.cc:237] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 2.17, Last appended by leader: 16, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } }
I20250627 13:54:26.981834 13528 raft_consensus.cc:357] T b265cfda4bfe48908659208ec8eeab2d P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:26.982146 13528 raft_consensus.cc:738] T b265cfda4bfe48908659208ec8eeab2d P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 956ff058e6d64579a1963aabdb6d466b, State: Initialized, Role: LEARNER
I20250627 13:54:26.982420 13528 consensus_queue.cc:260] T b265cfda4bfe48908659208ec8eeab2d P 956ff058e6d64579a1963aabdb6d466b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:26.984519 9973 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 17. Preceding OpId from leader: term: 2 index: 18. (index mismatch)
I20250627 13:54:26.985049 13434 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20250627 13:54:26.987087 7933 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 17. Preceding OpId from leader: term: 2 index: 18. (index mismatch)
I20250627 13:54:26.987751 13434 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250627 13:54:26.994259 13461 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 LEADER]: Committing config change with OpId 2.18: config changed from index 16 to 18, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:26.996306 13528 ts_tablet_manager.cc:1428] T b265cfda4bfe48908659208ec8eeab2d P 956ff058e6d64579a1963aabdb6d466b: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20250627 13:54:26.998905 7953 tablet_copy_service.cc:342] P a41a89ae8ed743edb2b5cf4490d8c854: Request end of tablet copy session 956ff058e6d64579a1963aabdb6d466b-b265cfda4bfe48908659208ec8eeab2d received from {username='slave'} at 127.0.103.70:60171
I20250627 13:54:26.999042 7953 tablet_copy_service.cc:434] P a41a89ae8ed743edb2b5cf4490d8c854: ending tablet copy session 956ff058e6d64579a1963aabdb6d466b-b265cfda4bfe48908659208ec8eeab2d on tablet b265cfda4bfe48908659208ec8eeab2d with peer 956ff058e6d64579a1963aabdb6d466b
I20250627 13:54:27.000316 7933 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Committing config change with OpId 2.18: config changed from index 16 to 18, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:27.007292 9973 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Committing config change with OpId 2.18: config changed from index 16 to 18, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:27.013105 7655 catalog_manager.cc:5582] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe reported cstate change: config changed from index 16 to 18, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New cstate: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:27.015923 7648 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet e3053b529a264e8697a1df9e17eeea81 with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:27.017814 8171 tablet_service.cc:1515] Processing DeleteTablet for tablet e3053b529a264e8697a1df9e17eeea81 with delete_type TABLET_DATA_TOMBSTONED (TS e4c831df4da841929d13d6c783099b0e not found in new config with opid_index 18) from {username='slave'} at 127.0.0.1:48616
I20250627 13:54:27.027854 7933 raft_consensus.cc:1273] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250627 13:54:27.028456 13461 consensus_queue.cc:1035] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:27.038472 13592 tablet_replica.cc:331] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: stopping tablet replica
I20250627 13:54:27.040479 13592 raft_consensus.cc:2241] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:27.040774 13592 raft_consensus.cc:2270] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:27.042464 13592 ts_tablet_manager.cc:1905] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:27.053226 9972 raft_consensus.cc:1273] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250627 13:54:27.054476 13461 consensus_queue.cc:1035] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:27.085584 7801 raft_consensus.cc:1273] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250627 13:54:27.086297 13434 consensus_queue.cc:1035] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:27.115893 13547 tablet_bootstrap.cc:492] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:27.116384 13547 tablet_bootstrap.cc:492] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e: Bootstrap complete.
I20250627 13:54:27.117019 13547 ts_tablet_manager.cc:1397] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.235s user 0.049s sys 0.014s
I20250627 13:54:27.118078 13547 raft_consensus.cc:357] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } }
I20250627 13:54:27.118322 13547 raft_consensus.cc:738] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: LEARNER
I20250627 13:54:27.118541 13547 consensus_queue.cc:260] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } }
I20250627 13:54:27.119151 13547 ts_tablet_manager.cc:1428] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:54:27.119267 9708 consensus_queue.cc:237] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:27.119760 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session e4c831df4da841929d13d6c783099b0e-9196497cb11f4ccbbd62695253d4c8c9 received from {username='slave'} at 127.0.103.68:59937
I20250627 13:54:27.119864 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session e4c831df4da841929d13d6c783099b0e-9196497cb11f4ccbbd62695253d4c8c9 on tablet 9196497cb11f4ccbbd62695253d4c8c9 with peer e4c831df4da841929d13d6c783099b0e
I20250627 13:54:27.122381 7933 raft_consensus.cc:1273] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250627 13:54:27.123631 9972 raft_consensus.cc:1273] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250627 13:54:27.125795 13434 consensus_queue.cc:1035] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:27.126214 13434 consensus_queue.cc:1035] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:27.129537 13434 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:27.131857 9972 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:27.135584 7648 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 40431e3bd7244904a4e65f59477bd21d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:27.135417 7933 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:27.139966 7658 catalog_manager.cc:5582] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe reported cstate change: config changed from index 8 to 10, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New cstate: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:27.145370 7782 tablet_service.cc:1515] Processing DeleteTablet for tablet 40431e3bd7244904a4e65f59477bd21d with delete_type TABLET_DATA_TOMBSTONED (TS 83574c1bf8ea45adaff939158f93c096 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:45826
I20250627 13:54:27.145539 13611 ts_tablet_manager.cc:927] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660: Initiating tablet copy from peer 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:54:27.146332 13614 tablet_replica.cc:331] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096: stopping tablet replica
I20250627 13:54:27.146994 13614 raft_consensus.cc:2241] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:27.147585 13614 raft_consensus.cc:2270] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:27.148802 13614 ts_tablet_manager.cc:1905] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:27.166090 13614 ts_tablet_manager.cc:1918] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20250627 13:54:27.166322 13614 log.cc:1199] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/wals/40431e3bd7244904a4e65f59477bd21d
I20250627 13:54:27.167366 7645 catalog_manager.cc:4928] TS 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467): tablet 40431e3bd7244904a4e65f59477bd21d (table rebalance_test_table_2 [id=6defd4a16efc47569b2193b71e4b2c4a]) successfully deleted
I20250627 13:54:27.179790 13611 tablet_copy_client.cc:323] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:37467
I20250627 13:54:27.181332 7822 tablet_copy_service.cc:140] P 83574c1bf8ea45adaff939158f93c096: Received BeginTabletCopySession request for tablet 1dc569a7b12d40058d073f7b81b2601c from peer 9cf479bdcd0242738d2a77db4ad9f660 ({username='slave'} at 127.0.103.67:60539)
I20250627 13:54:27.181510 7822 tablet_copy_service.cc:161] P 83574c1bf8ea45adaff939158f93c096: Beginning new tablet copy session on tablet 1dc569a7b12d40058d073f7b81b2601c from peer 9cf479bdcd0242738d2a77db4ad9f660 at {username='slave'} at 127.0.103.67:60539: session id = 9cf479bdcd0242738d2a77db4ad9f660-1dc569a7b12d40058d073f7b81b2601c
I20250627 13:54:27.184516 7822 tablet_copy_source_session.cc:215] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:27.186126 13611 ts_tablet_manager.cc:1905] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250627 13:54:27.186885 13592 ts_tablet_manager.cc:1918] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.17
I20250627 13:54:27.193284 13592 log.cc:1199] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal/wals/e3053b529a264e8697a1df9e17eeea81
I20250627 13:54:27.189847 13511 tablet_bootstrap.cc:492] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:27.194434 13511 tablet_bootstrap.cc:492] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e: Bootstrap complete.
I20250627 13:54:27.194818 13511 ts_tablet_manager.cc:1397] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.417s user 0.042s sys 0.015s
I20250627 13:54:27.198319 13511 raft_consensus.cc:357] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } }
I20250627 13:54:27.198554 13511 raft_consensus.cc:738] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: LEARNER
I20250627 13:54:27.199452 13511 consensus_queue.cc:260] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } }
I20250627 13:54:27.201824 13511 ts_tablet_manager.cc:1428] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250627 13:54:27.202781 10124 tablet_copy_service.cc:342] P dede81a8c0164427a61532d01b6fc157: Request end of tablet copy session e4c831df4da841929d13d6c783099b0e-5f176b66fdce4398be3492a5f3b4ea27 received from {username='slave'} at 127.0.103.68:39671
I20250627 13:54:27.203009 10124 tablet_copy_service.cc:434] P dede81a8c0164427a61532d01b6fc157: ending tablet copy session e4c831df4da841929d13d6c783099b0e-5f176b66fdce4398be3492a5f3b4ea27 on tablet 5f176b66fdce4398be3492a5f3b4ea27 with peer e4c831df4da841929d13d6c783099b0e
I20250627 13:54:27.218881 7648 catalog_manager.cc:4928] TS e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): tablet e3053b529a264e8697a1df9e17eeea81 (table rebalance_test_table_0 [id=d8bfc9268a4a4c2c92bc20d56ac73086]) successfully deleted
I20250627 13:54:27.227624 13611 ts_tablet_manager.cc:1918] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20250627 13:54:27.227931 13611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dc569a7b12d40058d073f7b81b2601c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:27.235682 13611 tablet_copy_client.cc:806] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:27.235977 13611 tablet_copy_client.cc:670] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:27.262789 13611 tablet_copy_client.cc:538] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:27.284252 13611 tablet_bootstrap.cc:492] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap starting.
I20250627 13:54:27.286381 13649 ts_tablet_manager.cc:927] T cff879c8e57d42c28440b3bd310f1ba9 P dede81a8c0164427a61532d01b6fc157: Initiating tablet copy from peer 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101)
I20250627 13:54:27.360540 8189 raft_consensus.cc:1215] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20250627 13:54:27.410630 8063 consensus_queue.cc:237] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } }
I20250627 13:54:27.413516 13611 tablet_bootstrap.cc:492] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:27.418582 13611 tablet_bootstrap.cc:492] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap complete.
I20250627 13:54:27.418861 13611 ts_tablet_manager.cc:1397] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660: Time spent bootstrapping tablet: real 0.135s user 0.049s sys 0.011s
I20250627 13:54:27.423677 13611 raft_consensus.cc:357] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: NON_VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: true } }
I20250627 13:54:27.423884 13611 raft_consensus.cc:738] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Initialized, Role: LEARNER
I20250627 13:54:27.429394 13611 consensus_queue.cc:260] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: NON_VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: true } }
I20250627 13:54:27.430027 13611 ts_tablet_manager.cc:1428] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250627 13:54:27.433923 7822 tablet_copy_service.cc:342] P 83574c1bf8ea45adaff939158f93c096: Request end of tablet copy session 9cf479bdcd0242738d2a77db4ad9f660-1dc569a7b12d40058d073f7b81b2601c received from {username='slave'} at 127.0.103.67:60539
I20250627 13:54:27.434069 7822 tablet_copy_service.cc:434] P 83574c1bf8ea45adaff939158f93c096: ending tablet copy session 9cf479bdcd0242738d2a77db4ad9f660-1dc569a7b12d40058d073f7b81b2601c on tablet 1dc569a7b12d40058d073f7b81b2601c with peer 9cf479bdcd0242738d2a77db4ad9f660
W20250627 13:54:27.442502 7998 consensus_peers.cc:487] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675): Couldn't send request to peer a41a89ae8ed743edb2b5cf4490d8c854. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:27.442775 8189 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20250627 13:54:27.443397 13651 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250627 13:54:27.454021 7801 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20250627 13:54:27.454876 13651 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250627 13:54:27.473109 13201 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:27.478677 9840 raft_consensus.cc:1215] T b265cfda4bfe48908659208ec8eeab2d P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20250627 13:54:27.488603 7801 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:27.505838 8190 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:27.507938 13649 tablet_copy_client.cc:323] T cff879c8e57d42c28440b3bd310f1ba9 P dede81a8c0164427a61532d01b6fc157: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.69:33101
I20250627 13:54:27.514699 7659 catalog_manager.cc:5582] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 15 to 16, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:27.522603 8190 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "da90822ff15e4e5da05114ef38d3970a"
dest_uuid: "e4c831df4da841929d13d6c783099b0e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60636
I20250627 13:54:27.523108 8190 raft_consensus.cc:604] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Received request to transfer leadership
I20250627 13:54:27.587687 8063 raft_consensus.cc:1215] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20250627 13:54:27.643693 13708 raft_consensus.cc:991] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e: : Instructing follower a41a89ae8ed743edb2b5cf4490d8c854 to start an election
I20250627 13:54:27.684759 9731 tablet_copy_service.cc:140] P 2a1282ebe92a4c9a92eed3b13f69012a: Received BeginTabletCopySession request for tablet cff879c8e57d42c28440b3bd310f1ba9 from peer dede81a8c0164427a61532d01b6fc157 ({username='slave'} at 127.0.103.72:45995)
I20250627 13:54:27.685014 9731 tablet_copy_service.cc:161] P 2a1282ebe92a4c9a92eed3b13f69012a: Beginning new tablet copy session on tablet cff879c8e57d42c28440b3bd310f1ba9 from peer dede81a8c0164427a61532d01b6fc157 at {username='slave'} at 127.0.103.72:45995: session id = dede81a8c0164427a61532d01b6fc157-cff879c8e57d42c28440b3bd310f1ba9
I20250627 13:54:27.689420 9731 tablet_copy_source_session.cc:215] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:27.692472 13649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cff879c8e57d42c28440b3bd310f1ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:27.704902 13649 tablet_copy_client.cc:806] T cff879c8e57d42c28440b3bd310f1ba9 P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:27.705156 13649 tablet_copy_client.cc:670] T cff879c8e57d42c28440b3bd310f1ba9 P dede81a8c0164427a61532d01b6fc157: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:27.708528 13649 tablet_copy_client.cc:538] T cff879c8e57d42c28440b3bd310f1ba9 P dede81a8c0164427a61532d01b6fc157: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:27.718102 13649 tablet_bootstrap.cc:492] T cff879c8e57d42c28440b3bd310f1ba9 P dede81a8c0164427a61532d01b6fc157: Bootstrap starting.
I20250627 13:54:27.736917 9822 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:27.741225 13651 raft_consensus.cc:1062] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER e4c831df4da841929d13d6c783099b0e to VOTER
I20250627 13:54:27.742100 13651 consensus_queue.cc:237] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } }
I20250627 13:54:27.748777 7911 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:27.752918 10083 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:27.760567 13714 raft_consensus.cc:1079] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Signalling peer a41a89ae8ed743edb2b5cf4490d8c854 to start an election
I20250627 13:54:27.765259 8189 raft_consensus.cc:1215] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
W20250627 13:54:27.780897 8129 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.69:33101, user_credentials={real_user=slave}} blocked reactor thread for 129609us
I20250627 13:54:27.787614 9691 tablet_service.cc:1430] Tablet server has 3 leaders and 0 scanners
I20250627 13:54:27.788729 8171 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
W20250627 13:54:27.809253 8128 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.72:35213, user_credentials={real_user=slave}} blocked reactor thread for 74081.2us
I20250627 13:54:27.810583 7933 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "da90822ff15e4e5da05114ef38d3970a"
dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
from {username='slave'} at 127.0.103.68:59649
I20250627 13:54:27.810778 7933 raft_consensus.cc:491] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:54:27.810911 7933 raft_consensus.cc:3058] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:27.814157 7933 raft_consensus.cc:513] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:27.814674 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:27.815371 8189 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da90822ff15e4e5da05114ef38d3970a" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: true dest_uuid: "e4c831df4da841929d13d6c783099b0e"
I20250627 13:54:27.815595 8189 raft_consensus.cc:3053] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Stepping down as leader of term 1
I20250627 13:54:27.815677 8189 raft_consensus.cc:738] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Running, Role: LEADER
I20250627 13:54:27.815922 8189 consensus_queue.cc:260] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:27.816262 8189 raft_consensus.cc:3058] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:27.818437 9840 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da90822ff15e4e5da05114ef38d3970a" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: true dest_uuid: "956ff058e6d64579a1963aabdb6d466b"
I20250627 13:54:27.818658 9840 raft_consensus.cc:3058] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:27.818715 7933 leader_election.cc:290] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 2 election: Requested vote from peers e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647), 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101)
I20250627 13:54:27.820489 8189 raft_consensus.cc:2466] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41a89ae8ed743edb2b5cf4490d8c854 in term 2.
I20250627 13:54:27.822425 9840 raft_consensus.cc:2466] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41a89ae8ed743edb2b5cf4490d8c854 in term 2.
I20250627 13:54:27.822638 8189 raft_consensus.cc:1273] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:27.822888 7866 leader_election.cc:304] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 956ff058e6d64579a1963aabdb6d466b, a41a89ae8ed743edb2b5cf4490d8c854, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:54:27.823195 13554 raft_consensus.cc:2802] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:54:27.823168 13678 consensus_queue.cc:1035] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:27.823320 13554 raft_consensus.cc:695] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 LEADER]: Becoming Leader. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Running, Role: LEADER
I20250627 13:54:27.823683 13554 consensus_queue.cc:237] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:27.831784 8041 tablet_service.cc:1430] Tablet server has 9 leaders and 0 scanners
I20250627 13:54:27.832212 9972 raft_consensus.cc:1273] T 9196497cb11f4ccbbd62695253d4c8c9 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:27.832890 13678 consensus_queue.cc:1035] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20250627 13:54:27.835186 10040 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.68:46139, user_credentials={real_user=slave}} blocked reactor thread for 69146.6us
I20250627 13:54:27.845700 9840 raft_consensus.cc:1273] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:27.848503 13651 consensus_queue.cc:1035] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:27.857756 9708 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da90822ff15e4e5da05114ef38d3970a" candidate_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: true dest_uuid: "2a1282ebe92a4c9a92eed3b13f69012a"
I20250627 13:54:27.861755 9708 raft_consensus.cc:3058] T da90822ff15e4e5da05114ef38d3970a P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:27.862098 7659 catalog_manager.cc:5582] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: term changed from 1 to 2, leader changed from e4c831df4da841929d13d6c783099b0e (127.0.103.68) to a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66). New cstate: current_term: 2 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:27.867142 13201 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:27.878650 9708 raft_consensus.cc:2466] T da90822ff15e4e5da05114ef38d3970a P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41a89ae8ed743edb2b5cf4490d8c854 in term 2.
I20250627 13:54:27.888698 9840 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:27.892160 8063 consensus_queue.cc:237] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } }
I20250627 13:54:27.897532 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:27.897910 9972 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:27.911780 8190 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:27.913782 7656 catalog_manager.cc:5582] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 10 to 11, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:27.917011 13546 raft_consensus.cc:1062] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854: attempting to promote NON_VOTER 956ff058e6d64579a1963aabdb6d466b to VOTER
I20250627 13:54:27.917883 13546 consensus_queue.cc:237] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:27.923126 9708 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:27.924034 13678 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20250627 13:54:27.926952 7999 consensus_peers.cc:487] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): Couldn't send request to peer e4c831df4da841929d13d6c783099b0e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:27.935544 9841 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:27.937760 13651 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:27.946261 13678 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
I20250627 13:54:27.950896 9841 raft_consensus.cc:1273] T b265cfda4bfe48908659208ec8eeab2d P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:27.951542 13554 consensus_queue.cc:1035] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:27.960479 7801 raft_consensus.cc:1273] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:27.961349 13554 consensus_queue.cc:1035] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:27.972172 9709 raft_consensus.cc:1273] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:27.972774 13554 consensus_queue.cc:1035] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:27.972654 9839 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
I20250627 13:54:27.978968 13688 raft_consensus.cc:1062] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096: attempting to promote NON_VOTER 9cf479bdcd0242738d2a77db4ad9f660 to VOTER
I20250627 13:54:27.979925 13688 consensus_queue.cc:237] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } }
I20250627 13:54:28.002921 13554 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:28.011049 7658 catalog_manager.cc:5582] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index 10 to 11, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:28.028934 9710 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
I20250627 13:54:28.036758 7658 catalog_manager.cc:5582] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: config changed from index 9 to 10, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
I20250627 13:54:28.038331 8063 raft_consensus.cc:1273] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEARNER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:28.041800 13760 consensus_queue.cc:1035] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:28.042964 9838 raft_consensus.cc:1273] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:28.044945 13760 consensus_queue.cc:1035] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:28.032456 7931 consensus_queue.cc:237] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:28.051875 9841 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
W20250627 13:54:28.025840 7868 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.72:35213, user_credentials={real_user=slave}} blocked reactor thread for 66572.5us
I20250627 13:54:28.054502 9710 raft_consensus.cc:1273] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250627 13:54:28.059751 13546 consensus_queue.cc:1035] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:28.060909 9708 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:28.061223 7801 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:28.076021 13759 ts_tablet_manager.cc:927] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
W20250627 13:54:28.078469 7999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.71:37859, user_credentials={real_user=slave}} blocked reactor thread for 43831.1us
I20250627 13:54:28.080919 7933 raft_consensus.cc:1273] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:28.082293 13534 consensus_queue.cc:1035] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:28.086787 8063 consensus_queue.cc:237] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } }
I20250627 13:54:28.087666 13534 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } }
I20250627 13:54:28.095543 8190 raft_consensus.cc:1273] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250627 13:54:28.097033 13678 consensus_queue.cc:1035] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:28.099033 9839 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } }
I20250627 13:54:28.102042 7656 catalog_manager.cc:5582] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 10 to 11, 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:28.102705 8063 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } }
I20250627 13:54:28.103632 13759 tablet_copy_client.cc:323] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:28.109922 13678 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:28.112397 8190 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:28.118443 7656 catalog_manager.cc:5582] T 9196497cb11f4ccbbd62695253d4c8c9 P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 11 to 12, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:28.111073 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet 644985e3086d4614b6598a727cb60c1d from peer a41a89ae8ed743edb2b5cf4490d8c854 ({username='slave'} at 127.0.103.66:40009)
I20250627 13:54:28.119170 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet 644985e3086d4614b6598a727cb60c1d from peer a41a89ae8ed743edb2b5cf4490d8c854 at {username='slave'} at 127.0.103.66:40009: session id = a41a89ae8ed743edb2b5cf4490d8c854-644985e3086d4614b6598a727cb60c1d
I20250627 13:54:28.122186 8082 tablet_copy_source_session.cc:215] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:28.123564 13759 ts_tablet_manager.cc:1905] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250627 13:54:28.127815 9972 raft_consensus.cc:1273] T 9196497cb11f4ccbbd62695253d4c8c9 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250627 13:54:28.128439 13651 consensus_queue.cc:1035] T 9196497cb11f4ccbbd62695253d4c8c9 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:28.132179 9822 tablet_service.cc:1515] Processing DeleteTablet for tablet 9196497cb11f4ccbbd62695253d4c8c9 with delete_type TABLET_DATA_TOMBSTONED (TS 956ff058e6d64579a1963aabdb6d466b not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:55228
I20250627 13:54:28.136420 9840 raft_consensus.cc:1273] T b265cfda4bfe48908659208ec8eeab2d P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250627 13:54:28.137740 13554 consensus_queue.cc:1035] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:28.150261 7933 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } }
I20250627 13:54:28.152944 13375 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:28.156554 7647 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet b265cfda4bfe48908659208ec8eeab2d with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:28.169915 13759 ts_tablet_manager.cc:1918] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20250627 13:54:28.170295 13759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 644985e3086d4614b6598a727cb60c1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:28.170740 9841 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:28.187124 9709 raft_consensus.cc:2953] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:28.197623 9972 raft_consensus.cc:2953] T 9196497cb11f4ccbbd62695253d4c8c9 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:28.199186 7659 catalog_manager.cc:5582] T b265cfda4bfe48908659208ec8eeab2d P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: config changed from index 11 to 12, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:28.205406 7782 tablet_service.cc:1515] Processing DeleteTablet for tablet b265cfda4bfe48908659208ec8eeab2d with delete_type TABLET_DATA_TOMBSTONED (TS 83574c1bf8ea45adaff939158f93c096 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:45826
I20250627 13:54:28.212421 13777 tablet_replica.cc:331] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b: stopping tablet replica
I20250627 13:54:28.212786 13777 raft_consensus.cc:2241] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:28.213136 13777 raft_consensus.cc:2270] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:28.214416 13777 ts_tablet_manager.cc:1905] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:28.233556 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9196497cb11f4ccbbd62695253d4c8c9 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:28.250015 13759 tablet_copy_client.cc:806] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:28.250370 13759 tablet_copy_client.cc:670] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:28.255543 13786 tablet_replica.cc:331] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096: stopping tablet replica
I20250627 13:54:28.255892 13786 raft_consensus.cc:2241] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:28.256268 13786 raft_consensus.cc:2270] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:28.292049 13759 tablet_copy_client.cc:538] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:28.305624 9691 tablet_service.cc:1430] Tablet server has 3 leaders and 0 scanners
I20250627 13:54:28.306277 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:28.307530 13786 ts_tablet_manager.cc:1905] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:28.328352 13786 ts_tablet_manager.cc:1918] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20250627 13:54:28.328524 13786 log.cc:1199] T b265cfda4bfe48908659208ec8eeab2d P 83574c1bf8ea45adaff939158f93c096: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/wals/b265cfda4bfe48908659208ec8eeab2d
I20250627 13:54:28.330528 7645 catalog_manager.cc:4928] TS 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467): tablet b265cfda4bfe48908659208ec8eeab2d (table rebalance_test_table_0 [id=d8bfc9268a4a4c2c92bc20d56ac73086]) successfully deleted
I20250627 13:54:28.332463 8041 tablet_service.cc:1430] Tablet server has 9 leaders and 0 scanners
I20250627 13:54:28.337091 13759 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:54:28.350039 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:28.359719 13777 ts_tablet_manager.cc:1918] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20250627 13:54:28.359977 13777 log.cc:1199] T 9196497cb11f4ccbbd62695253d4c8c9 P 956ff058e6d64579a1963aabdb6d466b: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal/wals/9196497cb11f4ccbbd62695253d4c8c9
I20250627 13:54:28.361871 9841 raft_consensus.cc:1273] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250627 13:54:28.374017 13554 consensus_queue.cc:1035] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:28.378460 7645 catalog_manager.cc:4928] TS 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): tablet 9196497cb11f4ccbbd62695253d4c8c9 (table rebalance_test_table_4 [id=4d92d465970b4ec4b5eace263b25362a]) successfully deleted
I20250627 13:54:28.409085 7911 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:28.414628 13649 tablet_bootstrap.cc:492] T cff879c8e57d42c28440b3bd310f1ba9 P dede81a8c0164427a61532d01b6fc157: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:28.426133 13649 tablet_bootstrap.cc:492] T cff879c8e57d42c28440b3bd310f1ba9 P dede81a8c0164427a61532d01b6fc157: Bootstrap complete.
I20250627 13:54:28.426584 13649 ts_tablet_manager.cc:1397] T cff879c8e57d42c28440b3bd310f1ba9 P dede81a8c0164427a61532d01b6fc157: Time spent bootstrapping tablet: real 0.709s user 0.059s sys 0.003s
I20250627 13:54:28.427588 13649 raft_consensus.cc:357] T cff879c8e57d42c28440b3bd310f1ba9 P dede81a8c0164427a61532d01b6fc157 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:28.427913 9708 raft_consensus.cc:1273] T da90822ff15e4e5da05114ef38d3970a P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250627 13:54:28.427944 13649 raft_consensus.cc:738] T cff879c8e57d42c28440b3bd310f1ba9 P dede81a8c0164427a61532d01b6fc157 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: dede81a8c0164427a61532d01b6fc157, State: Initialized, Role: LEARNER
I20250627 13:54:28.428256 13649 consensus_queue.cc:260] T cff879c8e57d42c28440b3bd310f1ba9 P dede81a8c0164427a61532d01b6fc157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } }
I20250627 13:54:28.428923 13554 consensus_queue.cc:1035] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:28.435869 8190 raft_consensus.cc:1273] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250627 13:54:28.436584 13554 consensus_queue.cc:1035] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:28.472314 10083 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:28.492535 13812 raft_consensus.cc:1062] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157: attempting to promote NON_VOTER e4c831df4da841929d13d6c783099b0e to VOTER
I20250627 13:54:28.526890 7933 consensus_queue.cc:237] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 10, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } }
I20250627 13:54:28.530160 9709 raft_consensus.cc:1273] T da90822ff15e4e5da05114ef38d3970a P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250627 13:54:28.534240 9822 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:28.534374 13778 consensus_queue.cc:1035] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250627 13:54:28.537509 13554 raft_consensus.cc:2953] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 10 to 12, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:28.544232 7647 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet da90822ff15e4e5da05114ef38d3970a with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:28.546665 9709 raft_consensus.cc:2953] T da90822ff15e4e5da05114ef38d3970a P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 10 to 12, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:28.546813 7658 catalog_manager.cc:5582] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index 10 to 12, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New cstate: current_term: 2 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:28.554718 8171 tablet_service.cc:1515] Processing DeleteTablet for tablet da90822ff15e4e5da05114ef38d3970a with delete_type TABLET_DATA_TOMBSTONED (TS e4c831df4da841929d13d6c783099b0e not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:48616
I20250627 13:54:28.555109 13824 ts_tablet_manager.cc:927] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:28.555544 9840 raft_consensus.cc:1273] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250627 13:54:28.557272 13778 consensus_queue.cc:1035] T da90822ff15e4e5da05114ef38d3970a P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:28.559331 9840 raft_consensus.cc:2953] T da90822ff15e4e5da05114ef38d3970a P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 10 to 12, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } }
I20250627 13:54:28.566422 13812 consensus_queue.cc:237] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } }
I20250627 13:54:28.567176 8172 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:28.586159 13824 tablet_copy_client.cc:323] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:28.586988 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet e2bd060a4a9e4765b45b0242ae56e489 from peer e4c831df4da841929d13d6c783099b0e ({username='slave'} at 127.0.103.68:59937)
I20250627 13:54:28.587134 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet e2bd060a4a9e4765b45b0242ae56e489 from peer e4c831df4da841929d13d6c783099b0e at {username='slave'} at 127.0.103.68:59937: session id = e4c831df4da841929d13d6c783099b0e-e2bd060a4a9e4765b45b0242ae56e489
I20250627 13:54:28.590070 8082 tablet_copy_source_session.cc:215] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:28.597496 13829 tablet_replica.cc:331] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e: stopping tablet replica
I20250627 13:54:28.605038 13829 raft_consensus.cc:2241] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:28.605304 13829 pending_rounds.cc:70] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e: Trying to abort 1 pending ops.
I20250627 13:54:28.605429 13829 pending_rounds.cc:77] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e: Aborting op as it isn't in flight: id { term: 2 index: 11 } timestamp: 7172228988209057792 op_type: NO_OP noop_request { }
I20250627 13:54:28.605665 13829 raft_consensus.cc:2887] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250627 13:54:28.606606 13829 raft_consensus.cc:2270] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:28.598165 13824 ts_tablet_manager.cc:1905] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250627 13:54:28.614447 13829 ts_tablet_manager.cc:1905] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:28.634218 8063 raft_consensus.cc:1273] T 5f176b66fdce4398be3492a5f3b4ea27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Refusing update from remote peer dede81a8c0164427a61532d01b6fc157: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250627 13:54:28.635329 7801 raft_consensus.cc:1273] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Refusing update from remote peer dede81a8c0164427a61532d01b6fc157: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250627 13:54:28.635922 13383 consensus_queue.cc:1035] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:28.641935 8190 raft_consensus.cc:1273] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 2 LEARNER]: Refusing update from remote peer dede81a8c0164427a61532d01b6fc157: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250627 13:54:28.654700 13824 ts_tablet_manager.cc:1918] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20250627 13:54:28.659926 13383 consensus_queue.cc:1035] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:28.660571 13383 consensus_queue.cc:1035] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:28.663842 13824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2bd060a4a9e4765b45b0242ae56e489. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:28.664916 13829 ts_tablet_manager.cc:1918] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.11
I20250627 13:54:28.665730 13829 log.cc:1199] T da90822ff15e4e5da05114ef38d3970a P e4c831df4da841929d13d6c783099b0e: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal/wals/da90822ff15e4e5da05114ef38d3970a
I20250627 13:54:28.670578 7648 catalog_manager.cc:4928] TS e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): tablet da90822ff15e4e5da05114ef38d3970a (table rebalance_test_table_2 [id=6defd4a16efc47569b2193b71e4b2c4a]) successfully deleted
I20250627 13:54:28.681188 13649 ts_tablet_manager.cc:1428] T cff879c8e57d42c28440b3bd310f1ba9 P dede81a8c0164427a61532d01b6fc157: Time spent starting tablet: real 0.254s user 0.106s sys 0.102s
I20250627 13:54:28.682358 9731 tablet_copy_service.cc:342] P 2a1282ebe92a4c9a92eed3b13f69012a: Request end of tablet copy session dede81a8c0164427a61532d01b6fc157-cff879c8e57d42c28440b3bd310f1ba9 received from {username='slave'} at 127.0.103.72:45995
I20250627 13:54:28.682610 9731 tablet_copy_service.cc:434] P 2a1282ebe92a4c9a92eed3b13f69012a: ending tablet copy session dede81a8c0164427a61532d01b6fc157-cff879c8e57d42c28440b3bd310f1ba9 on tablet cff879c8e57d42c28440b3bd310f1ba9 with peer dede81a8c0164427a61532d01b6fc157
I20250627 13:54:28.683147 13824 tablet_copy_client.cc:806] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:28.685989 13824 tablet_copy_client.cc:670] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:28.690613 13824 tablet_copy_client.cc:538] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:28.699381 13759 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:28.700186 13824 tablet_bootstrap.cc:492] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
I20250627 13:54:28.700628 13759 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap complete.
I20250627 13:54:28.705964 13759 ts_tablet_manager.cc:1397] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.369s user 0.055s sys 0.018s
I20250627 13:54:28.706795 13759 raft_consensus.cc:357] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } }
I20250627 13:54:28.707091 13759 raft_consensus.cc:738] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: LEARNER
I20250627 13:54:28.707315 13759 consensus_queue.cc:260] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } }
I20250627 13:54:28.707962 13759 ts_tablet_manager.cc:1428] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:54:28.708662 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session a41a89ae8ed743edb2b5cf4490d8c854-644985e3086d4614b6598a727cb60c1d received from {username='slave'} at 127.0.103.66:40009
I20250627 13:54:28.708781 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session a41a89ae8ed743edb2b5cf4490d8c854-644985e3086d4614b6598a727cb60c1d on tablet 644985e3086d4614b6598a727cb60c1d with peer a41a89ae8ed743edb2b5cf4490d8c854
I20250627 13:54:28.711367 13383 raft_consensus.cc:2953] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:28.719512 7658 catalog_manager.cc:5582] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 reported cstate change: config changed from index 8 to 9, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "dede81a8c0164427a61532d01b6fc157" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:28.724560 7801 raft_consensus.cc:2953] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:28.724638 8063 raft_consensus.cc:2953] T 5f176b66fdce4398be3492a5f3b4ea27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:28.730782 10102 consensus_queue.cc:237] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } }
I20250627 13:54:28.732486 8189 raft_consensus.cc:1273] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Refusing update from remote peer dede81a8c0164427a61532d01b6fc157: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250627 13:54:28.732720 8063 raft_consensus.cc:1273] T 5f176b66fdce4398be3492a5f3b4ea27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Refusing update from remote peer dede81a8c0164427a61532d01b6fc157: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250627 13:54:28.733222 13383 consensus_queue.cc:1035] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:28.735332 13383 consensus_queue.cc:1035] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:28.735878 8190 raft_consensus.cc:2953] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:28.749862 13818 raft_consensus.cc:2953] T 5f176b66fdce4398be3492a5f3b4ea27 P dede81a8c0164427a61532d01b6fc157 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:28.750504 8189 raft_consensus.cc:2953] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:28.773309 7655 catalog_manager.cc:5582] T 5f176b66fdce4398be3492a5f3b4ea27 P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 9 to 10, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New cstate: current_term: 2 leader_uuid: "dede81a8c0164427a61532d01b6fc157" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:28.775789 7647 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 5f176b66fdce4398be3492a5f3b4ea27 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:28.788552 7782 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f176b66fdce4398be3492a5f3b4ea27 with delete_type TABLET_DATA_TOMBSTONED (TS 83574c1bf8ea45adaff939158f93c096 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:45826
I20250627 13:54:28.788936 13786 tablet_replica.cc:331] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096: stopping tablet replica
I20250627 13:54:28.789115 13786 raft_consensus.cc:2241] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:28.789289 13786 raft_consensus.cc:2270] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:28.790249 13786 ts_tablet_manager.cc:1905] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:28.790230 8063 raft_consensus.cc:2953] T 5f176b66fdce4398be3492a5f3b4ea27 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:28.802219 13786 ts_tablet_manager.cc:1918] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20250627 13:54:28.802424 13786 log.cc:1199] T 5f176b66fdce4398be3492a5f3b4ea27 P 83574c1bf8ea45adaff939158f93c096: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/wals/5f176b66fdce4398be3492a5f3b4ea27
I20250627 13:54:28.803316 7645 catalog_manager.cc:4928] TS 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467): tablet 5f176b66fdce4398be3492a5f3b4ea27 (table rebalance_test_table_2 [id=6defd4a16efc47569b2193b71e4b2c4a]) successfully deleted
I20250627 13:54:28.850934 10102 raft_consensus.cc:1215] T cff879c8e57d42c28440b3bd310f1ba9 P dede81a8c0164427a61532d01b6fc157 [term 2 LEARNER]: Deduplicated request from leader. Original: 1.11->[2.12-2.12] Dedup: 2.12->[]
I20250627 13:54:28.895045 9691 tablet_service.cc:1430] Tablet server has 3 leaders and 0 scanners
I20250627 13:54:28.898872 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:28.912074 13824 tablet_bootstrap.cc:492] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:28.917098 7782 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:28.920570 13824 tablet_bootstrap.cc:492] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e: Bootstrap complete.
I20250627 13:54:28.921308 13824 ts_tablet_manager.cc:1397] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.227s user 0.053s sys 0.013s
I20250627 13:54:28.922405 8063 consensus_queue.cc:237] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } }
I20250627 13:54:28.923460 13824 raft_consensus.cc:357] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } }
W20250627 13:54:28.924968 7999 consensus_peers.cc:487] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): Couldn't send request to peer e4c831df4da841929d13d6c783099b0e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2eb537d49804e1b8ec5a217b1564988. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:28.924988 13824 raft_consensus.cc:738] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: LEARNER
I20250627 13:54:28.925621 13824 consensus_queue.cc:260] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } }
I20250627 13:54:28.926401 10102 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250627 13:54:28.926918 13764 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250627 13:54:28.927716 13824 ts_tablet_manager.cc:1428] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250627 13:54:28.930166 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session e4c831df4da841929d13d6c783099b0e-e2bd060a4a9e4765b45b0242ae56e489 received from {username='slave'} at 127.0.103.68:59937
I20250627 13:54:28.930266 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session e4c831df4da841929d13d6c783099b0e-e2bd060a4a9e4765b45b0242ae56e489 on tablet e2bd060a4a9e4765b45b0242ae56e489 with peer e4c831df4da841929d13d6c783099b0e
I20250627 13:54:28.931747 7801 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250627 13:54:28.932406 13651 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250627 13:54:28.938150 13651 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
I20250627 13:54:28.945098 7801 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
I20250627 13:54:28.945818 10102 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
I20250627 13:54:28.950737 7655 catalog_manager.cc:5582] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 12 to 13, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
I20250627 13:54:28.969133 9822 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:28.974601 7933 raft_consensus.cc:1215] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.15->[1.16-1.16] Dedup: 1.16->[]
I20250627 13:54:28.994242 8041 tablet_service.cc:1430] Tablet server has 9 leaders and 0 scanners
I20250627 13:54:29.000012 7911 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:29.065865 8189 raft_consensus.cc:1215] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20250627 13:54:29.073182 7933 consensus_queue.cc:237] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:29.079154 8171 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:29.115360 10083 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:29.135557 7801 consensus_queue.cc:237] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:29.137925 9972 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20250627 13:54:29.138136 7735 consensus_peers.cc:487] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 897206e017ff4305a8cc9092e30f46b6. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:29.138532 13761 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20250627 13:54:29.150321 7869 consensus_peers.cc:487] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 -> Peer 6686c6c365bf4f4582058890161676fe (127.0.103.71:37859): Couldn't send request to peer 6686c6c365bf4f4582058890161676fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83cfc975906745a09c5e1dcfac812ae2. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:29.152194 10102 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:29.152819 13912 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:29.166500 10104 raft_consensus.cc:1273] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250627 13:54:29.166574 9840 raft_consensus.cc:1273] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250627 13:54:29.166986 13904 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:29.167268 13901 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:29.172490 9972 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:29.161813 13912 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:29.180511 7658 catalog_manager.cc:5582] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe reported cstate change: config changed from index 9 to 10, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:29.180759 10103 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:29.234978 13901 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:29.237262 10102 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:29.244210 7655 catalog_manager.cc:5582] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157 reported cstate change: config changed from index 10 to 11, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New cstate: current_term: 2 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:29.258315 9840 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } } }
I20250627 13:54:29.260952 13603 raft_consensus.cc:1062] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a: attempting to promote NON_VOTER dede81a8c0164427a61532d01b6fc157 to VOTER
I20250627 13:54:29.262346 13603 consensus_queue.cc:237] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } }
I20250627 13:54:29.265915 10102 raft_consensus.cc:1273] T cff879c8e57d42c28440b3bd310f1ba9 P dede81a8c0164427a61532d01b6fc157 [term 2 LEARNER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250627 13:54:29.266710 13871 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250627 13:54:29.270115 8063 raft_consensus.cc:1273] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250627 13:54:29.270702 7933 raft_consensus.cc:1273] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250627 13:54:29.275064 13603 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250627 13:54:29.277709 13603 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250627 13:54:29.280207 13871 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 11 to 13, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:29.281252 8063 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:29.287729 7658 catalog_manager.cc:5582] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 11 to 13, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:29.293931 10102 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, dede81a8c0164427a61532d01b6fc157 (127.0.103.72) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:29.298270 7801 consensus_queue.cc:237] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } }
I20250627 13:54:29.323875 9709 consensus_queue.cc:237] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } }
I20250627 13:54:29.326139 10102 raft_consensus.cc:1273] T cff879c8e57d42c28440b3bd310f1ba9 P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250627 13:54:29.326747 13867 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250627 13:54:29.327309 8063 raft_consensus.cc:1273] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250627 13:54:29.328128 13867 consensus_queue.cc:1035] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250627 13:54:29.330399 13603 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:29.330857 10102 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
W20250627 13:54:29.334549 7737 consensus_peers.cc:487] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 -> Peer a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675): Couldn't send request to peer a41a89ae8ed743edb2b5cf4490d8c854. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:29.334801 9972 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:29.335148 9709 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250627 13:54:29.335347 13912 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:29.335636 13919 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:29.337448 7658 catalog_manager.cc:5582] T cff879c8e57d42c28440b3bd310f1ba9 P dede81a8c0164427a61532d01b6fc157 reported cstate change: config changed from index 13 to 14, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New cstate: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:29.339371 7648 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet cff879c8e57d42c28440b3bd310f1ba9 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:29.342555 8063 raft_consensus.cc:2953] T cff879c8e57d42c28440b3bd310f1ba9 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } }
I20250627 13:54:29.343511 7911 tablet_service.cc:1515] Processing DeleteTablet for tablet cff879c8e57d42c28440b3bd310f1ba9 with delete_type TABLET_DATA_TOMBSTONED (TS a41a89ae8ed743edb2b5cf4490d8c854 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:59302
I20250627 13:54:29.356593 13919 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:29.362140 9972 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:29.370378 7658 catalog_manager.cc:5582] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 9 to 10, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:29.371551 13824 ts_tablet_manager.cc:927] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:29.379348 9709 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:29.394889 13824 tablet_copy_client.cc:323] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:29.395463 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet d2eb537d49804e1b8ec5a217b1564988 from peer e4c831df4da841929d13d6c783099b0e ({username='slave'} at 127.0.103.68:59937)
I20250627 13:54:29.395607 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet d2eb537d49804e1b8ec5a217b1564988 from peer e4c831df4da841929d13d6c783099b0e at {username='slave'} at 127.0.103.68:59937: session id = e4c831df4da841929d13d6c783099b0e-d2eb537d49804e1b8ec5a217b1564988
I20250627 13:54:29.397343 9709 consensus_queue.cc:237] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 16, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } }
I20250627 13:54:29.398440 8082 tablet_copy_source_session.cc:215] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:29.399505 13959 tablet_replica.cc:331] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854: stopping tablet replica
I20250627 13:54:29.399763 13959 raft_consensus.cc:2241] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:29.400063 13959 pending_rounds.cc:70] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854: Trying to abort 1 pending ops.
I20250627 13:54:29.400105 13959 pending_rounds.cc:77] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854: Aborting op as it isn't in flight: id { term: 2 index: 13 } timestamp: 7172228994095194112 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "cff879c8e57d42c28440b3bd310f1ba9" old_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: NON_VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: true } } } new_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } } }
I20250627 13:54:29.400486 13959 raft_consensus.cc:2915] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Aborting config change with OpId 2.13: Aborted: Op aborted
I20250627 13:54:29.400955 13959 raft_consensus.cc:2887] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250627 13:54:29.401050 13959 raft_consensus.cc:2270] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:29.402202 13824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2eb537d49804e1b8ec5a217b1564988. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:29.402211 13959 ts_tablet_manager.cc:1905] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:29.416878 13959 ts_tablet_manager.cc:1918] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20250627 13:54:29.417028 13959 log.cc:1199] T cff879c8e57d42c28440b3bd310f1ba9 P a41a89ae8ed743edb2b5cf4490d8c854: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/wals/cff879c8e57d42c28440b3bd310f1ba9
I20250627 13:54:29.417999 7647 catalog_manager.cc:4928] TS a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675): tablet cff879c8e57d42c28440b3bd310f1ba9 (table rebalance_test_table_4 [id=4d92d465970b4ec4b5eace263b25362a]) successfully deleted
I20250627 13:54:29.466568 7933 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 18. Preceding OpId from leader: term: 2 index: 19. (index mismatch)
I20250627 13:54:29.466660 9972 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 18. Preceding OpId from leader: term: 2 index: 19. (index mismatch)
I20250627 13:54:29.467231 13949 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
W20250627 13:54:29.468122 9647 consensus_peers.cc:487] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a -> Peer e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): Couldn't send request to peer e4c831df4da841929d13d6c783099b0e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:29.468289 13949 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250627 13:54:29.468338 13824 tablet_copy_client.cc:806] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:29.468665 13824 tablet_copy_client.cc:670] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:29.475932 13824 tablet_copy_client.cc:538] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:29.478248 7801 consensus_queue.cc:237] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } }
I20250627 13:54:29.482029 10102 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20250627 13:54:29.483355 7737 consensus_peers.cc:487] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 -> Peer a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675): Couldn't send request to peer a41a89ae8ed743edb2b5cf4490d8c854. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:29.484035 9973 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:29.484210 13824 tablet_bootstrap.cc:492] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
W20250627 13:54:29.484313 7735 consensus_peers.cc:487] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 897206e017ff4305a8cc9092e30f46b6. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:29.484961 13761 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:29.485719 13761 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:29.488842 13761 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:29.490417 9973 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:29.491268 13949 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 LEADER]: Committing config change with OpId 2.19: config changed from index 18 to 19, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
I20250627 13:54:29.493760 7933 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Committing config change with OpId 2.19: config changed from index 18 to 19, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
I20250627 13:54:29.495330 10102 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:29.500092 7655 catalog_manager.cc:5582] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: config changed from index 18 to 19, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:29.510663 7655 catalog_manager.cc:5582] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe reported cstate change: config changed from index 10 to 11, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:29.545269 13888 raft_consensus.cc:1062] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 to VOTER
I20250627 13:54:29.546169 13888 consensus_queue.cc:237] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } }
I20250627 13:54:29.556445 9973 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Committing config change with OpId 2.19: config changed from index 18 to 19, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
I20250627 13:54:29.559008 7933 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250627 13:54:29.560297 13889 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250627 13:54:29.560505 8189 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250627 13:54:29.561028 13888 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250627 13:54:29.567202 7801 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250627 13:54:29.567665 13889 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250627 13:54:29.586520 13889 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.17: config changed from index 16 to 17, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:29.587476 8189 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:29.591104 7931 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:29.588951 13995 ts_tablet_manager.cc:927] T 83cfc975906745a09c5e1dcfac812ae2 P 6686c6c365bf4f4582058890161676fe: Initiating tablet copy from peer a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:54:29.597462 7658 catalog_manager.cc:5582] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 16 to 17, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:29.599448 9709 consensus_queue.cc:237] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } }
I20250627 13:54:29.607739 13995 tablet_copy_client.cc:323] T 83cfc975906745a09c5e1dcfac812ae2 P 6686c6c365bf4f4582058890161676fe: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:43675
I20250627 13:54:29.608534 8063 consensus_queue.cc:237] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 1.17, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } }
I20250627 13:54:29.610230 7931 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20250627 13:54:29.610930 13889 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250627 13:54:29.611141 8189 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20250627 13:54:29.613112 13889 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250627 13:54:29.614681 13889 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.18: config changed from index 17 to 18, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
W20250627 13:54:29.615834 9644 consensus_peers.cc:487] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a -> Peer 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467): Couldn't send request to peer 83574c1bf8ea45adaff939158f93c096. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:29.618432 7931 raft_consensus.cc:1273] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250627 13:54:29.618432 7933 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:29.619518 13867 consensus_queue.cc:1035] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:29.619885 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 644985e3086d4614b6598a727cb60c1d with cas_config_opid_index 17: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:29.622501 7658 catalog_manager.cc:5582] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 17 to 18, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:29.627666 7782 tablet_service.cc:1515] Processing DeleteTablet for tablet 644985e3086d4614b6598a727cb60c1d with delete_type TABLET_DATA_TOMBSTONED (TS 83574c1bf8ea45adaff939158f93c096 not found in new config with opid_index 18) from {username='slave'} at 127.0.0.1:45826
I20250627 13:54:29.632520 13888 raft_consensus.cc:1062] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER e4c831df4da841929d13d6c783099b0e to VOTER
I20250627 13:54:29.633356 13888 consensus_queue.cc:237] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } }
I20250627 13:54:29.634364 9972 raft_consensus.cc:1273] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250627 13:54:29.635042 13867 consensus_queue.cc:1035] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:29.641613 13867 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } }
I20250627 13:54:29.648047 7933 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } }
I20250627 13:54:29.649957 7656 catalog_manager.cc:5582] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: config changed from index 10 to 11, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New cstate: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:29.661842 8190 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:29.662379 7953 tablet_copy_service.cc:140] P a41a89ae8ed743edb2b5cf4490d8c854: Received BeginTabletCopySession request for tablet 83cfc975906745a09c5e1dcfac812ae2 from peer 6686c6c365bf4f4582058890161676fe ({username='slave'} at 127.0.103.71:56819)
I20250627 13:54:29.662531 7953 tablet_copy_service.cc:161] P a41a89ae8ed743edb2b5cf4490d8c854: Beginning new tablet copy session on tablet 83cfc975906745a09c5e1dcfac812ae2 from peer 6686c6c365bf4f4582058890161676fe at {username='slave'} at 127.0.103.71:56819: session id = 6686c6c365bf4f4582058890161676fe-83cfc975906745a09c5e1dcfac812ae2
I20250627 13:54:29.672221 7953 tablet_copy_source_session.cc:215] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:29.674026 13995 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83cfc975906745a09c5e1dcfac812ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:29.682137 14004 tablet_replica.cc:331] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096: stopping tablet replica
I20250627 13:54:29.683291 14004 raft_consensus.cc:2241] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:29.684152 13995 tablet_copy_client.cc:806] T 83cfc975906745a09c5e1dcfac812ae2 P 6686c6c365bf4f4582058890161676fe: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:29.684394 13995 tablet_copy_client.cc:670] T 83cfc975906745a09c5e1dcfac812ae2 P 6686c6c365bf4f4582058890161676fe: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:29.685233 14004 pending_rounds.cc:70] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096: Trying to abort 1 pending ops.
I20250627 13:54:29.686373 9709 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:29.686806 14004 pending_rounds.cc:77] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096: Aborting op as it isn't in flight: id { term: 1 index: 17 } timestamp: 7172228995258654720 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "644985e3086d4614b6598a727cb60c1d" old_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } } new_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } } }
I20250627 13:54:29.687002 13889 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:29.687485 9840 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:29.687721 8189 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:29.687793 14004 raft_consensus.cc:2915] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Aborting config change with OpId 1.17: Aborted: Op aborted
I20250627 13:54:29.688107 13888 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:29.688320 14004 raft_consensus.cc:2887] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250627 13:54:29.688516 14004 raft_consensus.cc:2270] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:29.690192 9973 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } }
I20250627 13:54:29.694482 13888 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:29.706862 14004 ts_tablet_manager.cc:1905] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:29.712092 13995 tablet_copy_client.cc:538] T 83cfc975906745a09c5e1dcfac812ae2 P 6686c6c365bf4f4582058890161676fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:29.712987 8190 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:29.715770 13995 tablet_bootstrap.cc:492] T 83cfc975906745a09c5e1dcfac812ae2 P 6686c6c365bf4f4582058890161676fe: Bootstrap starting.
I20250627 13:54:29.715989 13889 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:29.720026 7658 catalog_manager.cc:5582] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 10 to 11, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:29.721535 9842 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:29.732342 8063 consensus_queue.cc:237] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } }
I20250627 13:54:29.732017 9709 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:29.734469 8190 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250627 13:54:29.735427 14011 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:29.739732 9708 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250627 13:54:29.741145 14011 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:29.743703 8190 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:29.742867 14011 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:29.751816 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet e2bd060a4a9e4765b45b0242ae56e489 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:29.756374 7658 catalog_manager.cc:5582] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 11 to 12, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:29.759881 9708 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:29.791963 14004 ts_tablet_manager.cc:1918] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.17
I20250627 13:54:29.792114 14004 log.cc:1199] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/wals/644985e3086d4614b6598a727cb60c1d
I20250627 13:54:29.793041 7645 catalog_manager.cc:4928] TS 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467): tablet 644985e3086d4614b6598a727cb60c1d (table rebalance_test_table_1 [id=8961aa0df2f1488f82cd55cfa22ad214]) successfully deleted
I20250627 13:54:29.803608 13824 tablet_bootstrap.cc:492] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:29.804080 13824 tablet_bootstrap.cc:492] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e: Bootstrap complete.
I20250627 13:54:29.806666 13824 ts_tablet_manager.cc:1397] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.323s user 0.053s sys 0.013s
I20250627 13:54:29.811511 13824 raft_consensus.cc:357] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } }
I20250627 13:54:29.811769 13824 raft_consensus.cc:738] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: LEARNER
I20250627 13:54:29.812018 13824 consensus_queue.cc:260] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } }
I20250627 13:54:29.815387 13824 ts_tablet_manager.cc:1428] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250627 13:54:29.821142 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session e4c831df4da841929d13d6c783099b0e-d2eb537d49804e1b8ec5a217b1564988 received from {username='slave'} at 127.0.103.68:59937
I20250627 13:54:29.821285 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session e4c831df4da841929d13d6c783099b0e-d2eb537d49804e1b8ec5a217b1564988 on tablet d2eb537d49804e1b8ec5a217b1564988 with peer e4c831df4da841929d13d6c783099b0e
W20250627 13:54:29.832036 9775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.65:37467, user_credentials={real_user=slave}} blocked reactor thread for 93700.5us
I20250627 13:54:29.835355 9822 tablet_service.cc:1515] Processing DeleteTablet for tablet e2bd060a4a9e4765b45b0242ae56e489 with delete_type TABLET_DATA_TOMBSTONED (TS 956ff058e6d64579a1963aabdb6d466b not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:55228
I20250627 13:54:29.843758 14057 tablet_replica.cc:331] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b: stopping tablet replica
I20250627 13:54:29.844064 14057 raft_consensus.cc:2241] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:29.844358 14057 raft_consensus.cc:2270] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:29.845439 14057 ts_tablet_manager.cc:1905] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:29.862437 14057 ts_tablet_manager.cc:1918] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20250627 13:54:29.869416 14057 log.cc:1199] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal/wals/e2bd060a4a9e4765b45b0242ae56e489
I20250627 13:54:29.870383 7645 catalog_manager.cc:4928] TS 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): tablet e2bd060a4a9e4765b45b0242ae56e489 (table rebalance_test_table_6 [id=f83cab74056a4e7681a701169d1c1716]) successfully deleted
I20250627 13:54:29.873587 13995 tablet_bootstrap.cc:492] T 83cfc975906745a09c5e1dcfac812ae2 P 6686c6c365bf4f4582058890161676fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:29.874202 13995 tablet_bootstrap.cc:492] T 83cfc975906745a09c5e1dcfac812ae2 P 6686c6c365bf4f4582058890161676fe: Bootstrap complete.
I20250627 13:54:29.874569 13995 ts_tablet_manager.cc:1397] T 83cfc975906745a09c5e1dcfac812ae2 P 6686c6c365bf4f4582058890161676fe: Time spent bootstrapping tablet: real 0.159s user 0.060s sys 0.004s
I20250627 13:54:29.875526 13995 raft_consensus.cc:357] T 83cfc975906745a09c5e1dcfac812ae2 P 6686c6c365bf4f4582058890161676fe [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:29.878567 13995 raft_consensus.cc:738] T 83cfc975906745a09c5e1dcfac812ae2 P 6686c6c365bf4f4582058890161676fe [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 6686c6c365bf4f4582058890161676fe, State: Initialized, Role: LEARNER
I20250627 13:54:29.878952 13995 consensus_queue.cc:260] T 83cfc975906745a09c5e1dcfac812ae2 P 6686c6c365bf4f4582058890161676fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: NON_VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: true } }
I20250627 13:54:29.879805 13995 ts_tablet_manager.cc:1428] T 83cfc975906745a09c5e1dcfac812ae2 P 6686c6c365bf4f4582058890161676fe: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250627 13:54:29.880571 7953 tablet_copy_service.cc:342] P a41a89ae8ed743edb2b5cf4490d8c854: Request end of tablet copy session 6686c6c365bf4f4582058890161676fe-83cfc975906745a09c5e1dcfac812ae2 received from {username='slave'} at 127.0.103.71:56819
I20250627 13:54:29.880743 7953 tablet_copy_service.cc:434] P a41a89ae8ed743edb2b5cf4490d8c854: ending tablet copy session 6686c6c365bf4f4582058890161676fe-83cfc975906745a09c5e1dcfac812ae2 on tablet 83cfc975906745a09c5e1dcfac812ae2 with peer 6686c6c365bf4f4582058890161676fe
I20250627 13:54:29.903942 14068 ts_tablet_manager.cc:927] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854: Initiating tablet copy from peer 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:54:29.922714 14068 tablet_copy_client.cc:323] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:37467
I20250627 13:54:29.923445 7822 tablet_copy_service.cc:140] P 83574c1bf8ea45adaff939158f93c096: Received BeginTabletCopySession request for tablet 897206e017ff4305a8cc9092e30f46b6 from peer a41a89ae8ed743edb2b5cf4490d8c854 ({username='slave'} at 127.0.103.66:49677)
I20250627 13:54:29.923604 7822 tablet_copy_service.cc:161] P 83574c1bf8ea45adaff939158f93c096: Beginning new tablet copy session on tablet 897206e017ff4305a8cc9092e30f46b6 from peer a41a89ae8ed743edb2b5cf4490d8c854 at {username='slave'} at 127.0.103.66:49677: session id = a41a89ae8ed743edb2b5cf4490d8c854-897206e017ff4305a8cc9092e30f46b6
I20250627 13:54:29.924355 7800 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "1dc569a7b12d40058d073f7b81b2601c"
dest_uuid: "83574c1bf8ea45adaff939158f93c096"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59032
I20250627 13:54:29.924508 7800 raft_consensus.cc:604] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Received request to transfer leadership
I20250627 13:54:29.926885 7822 tablet_copy_source_session.cc:215] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:29.928120 14068 ts_tablet_manager.cc:1905] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250627 13:54:29.942018 14068 ts_tablet_manager.cc:1918] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20250627 13:54:29.946390 14068 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 897206e017ff4305a8cc9092e30f46b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:29.948819 14084 ts_tablet_manager.cc:927] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854: Initiating tablet copy from peer 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:54:29.963469 14068 tablet_copy_client.cc:806] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:29.963785 14068 tablet_copy_client.cc:670] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:29.964295 14084 tablet_copy_client.cc:323] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:37467
I20250627 13:54:29.965003 7821 tablet_copy_service.cc:140] P 83574c1bf8ea45adaff939158f93c096: Received BeginTabletCopySession request for tablet 87726824b05a4b968386f57d3a7a4aad from peer a41a89ae8ed743edb2b5cf4490d8c854 ({username='slave'} at 127.0.103.66:49677)
I20250627 13:54:29.965164 7821 tablet_copy_service.cc:161] P 83574c1bf8ea45adaff939158f93c096: Beginning new tablet copy session on tablet 87726824b05a4b968386f57d3a7a4aad from peer a41a89ae8ed743edb2b5cf4490d8c854 at {username='slave'} at 127.0.103.66:49677: session id = a41a89ae8ed743edb2b5cf4490d8c854-87726824b05a4b968386f57d3a7a4aad
I20250627 13:54:29.965734 14068 tablet_copy_client.cc:538] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:29.968044 7821 tablet_copy_source_session.cc:215] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:29.968950 14084 ts_tablet_manager.cc:1905] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250627 13:54:29.969147 14068 tablet_bootstrap.cc:492] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:54:29.976027 14084 ts_tablet_manager.cc:1918] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20250627 13:54:29.976454 14084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87726824b05a4b968386f57d3a7a4aad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:29.991071 14084 tablet_copy_client.cc:806] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:29.991348 14084 tablet_copy_client.cc:670] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:29.993222 14084 tablet_copy_client.cc:538] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:30.004438 14084 tablet_bootstrap.cc:492] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:54:30.035110 9972 raft_consensus.cc:1215] T 83cfc975906745a09c5e1dcfac812ae2 P 6686c6c365bf4f4582058890161676fe [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20250627 13:54:30.074340 13824 ts_tablet_manager.cc:927] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: Initiating tablet copy from peer 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101)
I20250627 13:54:30.089916 14112 ts_tablet_manager.cc:927] T 897206e017ff4305a8cc9092e30f46b6 P 956ff058e6d64579a1963aabdb6d466b: Initiating tablet copy from peer 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467)
I20250627 13:54:30.089995 13824 tablet_copy_client.cc:323] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.69:33101
I20250627 13:54:30.090768 9731 tablet_copy_service.cc:140] P 2a1282ebe92a4c9a92eed3b13f69012a: Received BeginTabletCopySession request for tablet e3053b529a264e8697a1df9e17eeea81 from peer e4c831df4da841929d13d6c783099b0e ({username='slave'} at 127.0.103.68:51963)
I20250627 13:54:30.090999 9731 tablet_copy_service.cc:161] P 2a1282ebe92a4c9a92eed3b13f69012a: Beginning new tablet copy session on tablet e3053b529a264e8697a1df9e17eeea81 from peer e4c831df4da841929d13d6c783099b0e at {username='slave'} at 127.0.103.68:51963: session id = e4c831df4da841929d13d6c783099b0e-e3053b529a264e8697a1df9e17eeea81
I20250627 13:54:30.092641 14068 tablet_bootstrap.cc:492] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:30.093166 14068 tablet_bootstrap.cc:492] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap complete.
I20250627 13:54:30.093452 14068 ts_tablet_manager.cc:1397] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.124s user 0.049s sys 0.021s
I20250627 13:54:30.094317 14068 raft_consensus.cc:357] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } }
I20250627 13:54:30.094552 14068 raft_consensus.cc:738] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: LEARNER
I20250627 13:54:30.094790 14068 consensus_queue.cc:260] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } }
I20250627 13:54:30.094877 9731 tablet_copy_source_session.cc:215] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:30.095448 14068 ts_tablet_manager.cc:1428] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.002s user 0.001s sys 0.004s
I20250627 13:54:30.096141 7821 tablet_copy_service.cc:342] P 83574c1bf8ea45adaff939158f93c096: Request end of tablet copy session a41a89ae8ed743edb2b5cf4490d8c854-897206e017ff4305a8cc9092e30f46b6 received from {username='slave'} at 127.0.103.66:49677
I20250627 13:54:30.096227 13824 ts_tablet_manager.cc:1905] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250627 13:54:30.096269 7821 tablet_copy_service.cc:434] P 83574c1bf8ea45adaff939158f93c096: ending tablet copy session a41a89ae8ed743edb2b5cf4490d8c854-897206e017ff4305a8cc9092e30f46b6 on tablet 897206e017ff4305a8cc9092e30f46b6 with peer a41a89ae8ed743edb2b5cf4490d8c854
I20250627 13:54:30.100203 14112 tablet_copy_client.cc:323] T 897206e017ff4305a8cc9092e30f46b6 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:37467
I20250627 13:54:30.100889 7821 tablet_copy_service.cc:140] P 83574c1bf8ea45adaff939158f93c096: Received BeginTabletCopySession request for tablet 897206e017ff4305a8cc9092e30f46b6 from peer 956ff058e6d64579a1963aabdb6d466b ({username='slave'} at 127.0.103.70:37961)
I20250627 13:54:30.101044 7821 tablet_copy_service.cc:161] P 83574c1bf8ea45adaff939158f93c096: Beginning new tablet copy session on tablet 897206e017ff4305a8cc9092e30f46b6 from peer 956ff058e6d64579a1963aabdb6d466b at {username='slave'} at 127.0.103.70:37961: session id = 956ff058e6d64579a1963aabdb6d466b-897206e017ff4305a8cc9092e30f46b6
I20250627 13:54:30.103317 7821 tablet_copy_source_session.cc:215] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:30.104403 13824 ts_tablet_manager.cc:1918] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.17
I20250627 13:54:30.104710 13824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3053b529a264e8697a1df9e17eeea81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:30.104995 14112 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 897206e017ff4305a8cc9092e30f46b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:30.108212 13824 tablet_copy_client.cc:806] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:30.108414 13824 tablet_copy_client.cc:670] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:30.110159 13824 tablet_copy_client.cc:538] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:30.114711 13824 tablet_bootstrap.cc:492] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
I20250627 13:54:30.122318 14112 tablet_copy_client.cc:806] T 897206e017ff4305a8cc9092e30f46b6 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:30.122599 14112 tablet_copy_client.cc:670] T 897206e017ff4305a8cc9092e30f46b6 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:30.124843 14112 tablet_copy_client.cc:538] T 897206e017ff4305a8cc9092e30f46b6 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:30.133551 14112 tablet_bootstrap.cc:492] T 897206e017ff4305a8cc9092e30f46b6 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap starting.
I20250627 13:54:30.154225 13977 raft_consensus.cc:991] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096: : Instructing follower 956ff058e6d64579a1963aabdb6d466b to start an election
I20250627 13:54:30.154362 13977 raft_consensus.cc:1079] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Signalling peer 956ff058e6d64579a1963aabdb6d466b to start an election
I20250627 13:54:30.155100 9840 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "1dc569a7b12d40058d073f7b81b2601c"
dest_uuid: "956ff058e6d64579a1963aabdb6d466b"
from {username='slave'} at 127.0.103.65:41001
I20250627 13:54:30.155323 9840 raft_consensus.cc:491] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:54:30.155434 9840 raft_consensus.cc:3058] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:30.158972 9840 raft_consensus.cc:513] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } }
I20250627 13:54:30.160881 7801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dc569a7b12d40058d073f7b81b2601c" candidate_uuid: "956ff058e6d64579a1963aabdb6d466b" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "83574c1bf8ea45adaff939158f93c096"
I20250627 13:54:30.161067 7801 raft_consensus.cc:3053] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Stepping down as leader of term 1
I20250627 13:54:30.161130 7801 raft_consensus.cc:738] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Running, Role: LEADER
I20250627 13:54:30.161386 7801 consensus_queue.cc:260] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } }
I20250627 13:54:30.161790 7801 raft_consensus.cc:3058] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:30.164646 7801 raft_consensus.cc:2466] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 956ff058e6d64579a1963aabdb6d466b in term 2.
I20250627 13:54:30.167862 7933 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dc569a7b12d40058d073f7b81b2601c" candidate_uuid: "956ff058e6d64579a1963aabdb6d466b" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
I20250627 13:54:30.168058 7933 raft_consensus.cc:3058] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:30.169214 14127 ts_tablet_manager.cc:927] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096: Initiating tablet copy from peer 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101)
I20250627 13:54:30.170909 7933 raft_consensus.cc:2466] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 956ff058e6d64579a1963aabdb6d466b in term 2.
I20250627 13:54:30.171398 9777 leader_election.cc:304] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, 956ff058e6d64579a1963aabdb6d466b, a41a89ae8ed743edb2b5cf4490d8c854; no voters:
I20250627 13:54:30.171778 13544 raft_consensus.cc:2802] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:54:30.171911 13544 raft_consensus.cc:695] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b [term 2 LEADER]: Becoming Leader. State: Replica: 956ff058e6d64579a1963aabdb6d466b, State: Running, Role: LEADER
I20250627 13:54:30.172297 13544 consensus_queue.cc:237] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } }
I20250627 13:54:30.172931 14127 tablet_copy_client.cc:323] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.69:33101
I20250627 13:54:30.173684 9731 tablet_copy_service.cc:140] P 2a1282ebe92a4c9a92eed3b13f69012a: Received BeginTabletCopySession request for tablet 40431e3bd7244904a4e65f59477bd21d from peer 83574c1bf8ea45adaff939158f93c096 ({username='slave'} at 127.0.103.65:56847)
I20250627 13:54:30.173858 9731 tablet_copy_service.cc:161] P 2a1282ebe92a4c9a92eed3b13f69012a: Beginning new tablet copy session on tablet 40431e3bd7244904a4e65f59477bd21d from peer 83574c1bf8ea45adaff939158f93c096 at {username='slave'} at 127.0.103.65:56847: session id = 83574c1bf8ea45adaff939158f93c096-40431e3bd7244904a4e65f59477bd21d
I20250627 13:54:30.176666 14084 tablet_bootstrap.cc:492] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:30.176914 9731 tablet_copy_source_session.cc:215] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:30.177062 14084 tablet_bootstrap.cc:492] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap complete.
I20250627 13:54:30.177317 14084 ts_tablet_manager.cc:1397] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.173s user 0.059s sys 0.008s
I20250627 13:54:30.178196 14084 raft_consensus.cc:357] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } }
I20250627 13:54:30.178483 14084 raft_consensus.cc:738] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: LEARNER
I20250627 13:54:30.178709 14084 consensus_queue.cc:260] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } }
I20250627 13:54:30.179508 14127 ts_tablet_manager.cc:1905] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250627 13:54:30.180212 14084 ts_tablet_manager.cc:1428] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20250627 13:54:30.180848 7821 tablet_copy_service.cc:342] P 83574c1bf8ea45adaff939158f93c096: Request end of tablet copy session a41a89ae8ed743edb2b5cf4490d8c854-87726824b05a4b968386f57d3a7a4aad received from {username='slave'} at 127.0.103.66:49677
I20250627 13:54:30.180936 7821 tablet_copy_service.cc:434] P 83574c1bf8ea45adaff939158f93c096: ending tablet copy session a41a89ae8ed743edb2b5cf4490d8c854-87726824b05a4b968386f57d3a7a4aad on tablet 87726824b05a4b968386f57d3a7a4aad with peer a41a89ae8ed743edb2b5cf4490d8c854
I20250627 13:54:30.187013 14127 ts_tablet_manager.cc:1918] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.9
I20250627 13:54:30.187269 14127 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40431e3bd7244904a4e65f59477bd21d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:30.189018 7656 catalog_manager.cc:5582] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b reported cstate change: term changed from 1 to 2, leader changed from 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) to 956ff058e6d64579a1963aabdb6d466b (127.0.103.70). New cstate: current_term: 2 leader_uuid: "956ff058e6d64579a1963aabdb6d466b" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:30.191074 14127 tablet_copy_client.cc:806] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:30.191316 14127 tablet_copy_client.cc:670] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:30.191798 9840 leader_election.cc:290] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b [CANDIDATE]: Term 2 election: Requested vote from peers 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467), a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675), 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:30.192605 8063 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dc569a7b12d40058d073f7b81b2601c" candidate_uuid: "956ff058e6d64579a1963aabdb6d466b" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660"
I20250627 13:54:30.192821 8063 raft_consensus.cc:3058] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:30.195865 8063 raft_consensus.cc:2466] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 956ff058e6d64579a1963aabdb6d466b in term 2.
I20250627 13:54:30.199258 14127 tablet_copy_client.cc:538] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:30.206976 14127 tablet_bootstrap.cc:492] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:54:30.346383 8190 raft_consensus.cc:1215] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13] Dedup: 1.13->[]
I20250627 13:54:30.423950 14127 tablet_bootstrap.cc:492] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:30.424377 14127 tablet_bootstrap.cc:492] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096: Bootstrap complete.
I20250627 13:54:30.424719 14127 ts_tablet_manager.cc:1397] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.218s user 0.058s sys 0.008s
I20250627 13:54:30.425727 14127 raft_consensus.cc:357] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } }
I20250627 13:54:30.425967 14127 raft_consensus.cc:738] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: LEARNER
I20250627 13:54:30.426199 14127 consensus_queue.cc:260] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } }
I20250627 13:54:30.426890 14127 ts_tablet_manager.cc:1428] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:54:30.427660 9731 tablet_copy_service.cc:342] P 2a1282ebe92a4c9a92eed3b13f69012a: Request end of tablet copy session 83574c1bf8ea45adaff939158f93c096-40431e3bd7244904a4e65f59477bd21d received from {username='slave'} at 127.0.103.65:56847
I20250627 13:54:30.427838 9731 tablet_copy_service.cc:434] P 2a1282ebe92a4c9a92eed3b13f69012a: ending tablet copy session 83574c1bf8ea45adaff939158f93c096-40431e3bd7244904a4e65f59477bd21d on tablet 40431e3bd7244904a4e65f59477bd21d with peer 83574c1bf8ea45adaff939158f93c096
I20250627 13:54:30.432870 7933 raft_consensus.cc:1215] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20250627 13:54:30.467672 7933 raft_consensus.cc:1215] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20250627 13:54:30.472325 13915 raft_consensus.cc:1062] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854: attempting to promote NON_VOTER 6686c6c365bf4f4582058890161676fe to VOTER
I20250627 13:54:30.473250 13915 consensus_queue.cc:237] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:30.489523 9972 raft_consensus.cc:1273] T 83cfc975906745a09c5e1dcfac812ae2 P 6686c6c365bf4f4582058890161676fe [term 2 LEARNER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250627 13:54:30.490084 14086 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:30.491029 13824 tablet_bootstrap.cc:492] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=19 overwritten=0 applied=19 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:30.491439 13824 tablet_bootstrap.cc:492] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: Bootstrap complete.
I20250627 13:54:30.491508 9840 raft_consensus.cc:1273] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250627 13:54:30.491680 13824 ts_tablet_manager.cc:1397] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.377s user 0.075s sys 0.029s
I20250627 13:54:30.494005 14086 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:30.499639 13824 raft_consensus.cc:357] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } }
I20250627 13:54:30.499851 13824 raft_consensus.cc:738] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: LEARNER
I20250627 13:54:30.500072 13824 consensus_queue.cc:260] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 19, Last appended: 2.19, Last appended by leader: 19, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } }
I20250627 13:54:30.501472 13824 ts_tablet_manager.cc:1428] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.010s user 0.002s sys 0.001s
I20250627 13:54:30.503425 9731 tablet_copy_service.cc:342] P 2a1282ebe92a4c9a92eed3b13f69012a: Request end of tablet copy session e4c831df4da841929d13d6c783099b0e-e3053b529a264e8697a1df9e17eeea81 received from {username='slave'} at 127.0.103.68:51963
I20250627 13:54:30.503602 9731 tablet_copy_service.cc:434] P 2a1282ebe92a4c9a92eed3b13f69012a: ending tablet copy session e4c831df4da841929d13d6c783099b0e-e3053b529a264e8697a1df9e17eeea81 on tablet e3053b529a264e8697a1df9e17eeea81 with peer e4c831df4da841929d13d6c783099b0e
I20250627 13:54:30.504882 14112 tablet_bootstrap.cc:492] T 897206e017ff4305a8cc9092e30f46b6 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:30.505353 14112 tablet_bootstrap.cc:492] T 897206e017ff4305a8cc9092e30f46b6 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap complete.
I20250627 13:54:30.505704 14112 ts_tablet_manager.cc:1397] T 897206e017ff4305a8cc9092e30f46b6 P 956ff058e6d64579a1963aabdb6d466b: Time spent bootstrapping tablet: real 0.372s user 0.044s sys 0.013s
I20250627 13:54:30.506582 14112 raft_consensus.cc:357] T 897206e017ff4305a8cc9092e30f46b6 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } }
I20250627 13:54:30.538914 14112 raft_consensus.cc:738] T 897206e017ff4305a8cc9092e30f46b6 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 956ff058e6d64579a1963aabdb6d466b, State: Initialized, Role: LEARNER
I20250627 13:54:30.539211 14112 consensus_queue.cc:260] T 897206e017ff4305a8cc9092e30f46b6 P 956ff058e6d64579a1963aabdb6d466b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } }
I20250627 13:54:30.539839 14112 ts_tablet_manager.cc:1428] T 897206e017ff4305a8cc9092e30f46b6 P 956ff058e6d64579a1963aabdb6d466b: Time spent starting tablet: real 0.034s user 0.000s sys 0.001s
I20250627 13:54:30.541553 7821 tablet_copy_service.cc:342] P 83574c1bf8ea45adaff939158f93c096: Request end of tablet copy session 956ff058e6d64579a1963aabdb6d466b-897206e017ff4305a8cc9092e30f46b6 received from {username='slave'} at 127.0.103.70:37961
I20250627 13:54:30.541704 7821 tablet_copy_service.cc:434] P 83574c1bf8ea45adaff939158f93c096: ending tablet copy session 956ff058e6d64579a1963aabdb6d466b-897206e017ff4305a8cc9092e30f46b6 on tablet 897206e017ff4305a8cc9092e30f46b6 with peer 956ff058e6d64579a1963aabdb6d466b
I20250627 13:54:30.543704 14086 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:30.551488 9972 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:30.551017 9840 raft_consensus.cc:1215] T 897206e017ff4305a8cc9092e30f46b6 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20250627 13:54:30.589221 7800 raft_consensus.cc:1273] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Refusing update from remote peer 956ff058e6d64579a1963aabdb6d466b: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250627 13:54:30.589982 13544 consensus_queue.cc:1035] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:30.592587 10102 raft_consensus.cc:1273] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250627 13:54:30.596982 9840 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:30.597921 13915 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:30.616192 7661 catalog_manager.cc:5582] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b reported cstate change: config changed from index 11 to 12, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:30.622174 8190 raft_consensus.cc:1215] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.18->[2.19-2.19] Dedup: 2.19->[]
W20250627 13:54:30.639456 7868 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.72:35213, user_credentials={real_user=slave}} blocked reactor thread for 46308us
I20250627 13:54:30.640172 7933 raft_consensus.cc:1273] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Refusing update from remote peer 956ff058e6d64579a1963aabdb6d466b: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250627 13:54:30.647008 13544 consensus_queue.cc:1035] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20250627 13:54:30.674880 9647 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.68:46139, user_credentials={real_user=slave}} blocked reactor thread for 48766.6us
I20250627 13:54:30.682497 10102 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 6686c6c365bf4f4582058890161676fe (127.0.103.71) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:30.690264 9822 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
I20250627 13:54:30.697094 7911 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:30.711651 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:30.718395 9691 tablet_service.cc:1430] Tablet server has 3 leaders and 0 scanners
I20250627 13:54:30.721341 7782 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
W20250627 13:54:30.743506 9775 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.65:37467, user_credentials={real_user=slave}} blocked reactor thread for 67517.7us
I20250627 13:54:30.756718 8063 raft_consensus.cc:1273] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Refusing update from remote peer 956ff058e6d64579a1963aabdb6d466b: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250627 13:54:30.765489 13544 consensus_queue.cc:1035] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:30.771404 7911 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:30.777010 7801 raft_consensus.cc:1215] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20250627 13:54:30.786343 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:30.789284 8041 tablet_service.cc:1430] Tablet server has 9 leaders and 0 scanners
I20250627 13:54:30.794102 9691 tablet_service.cc:1430] Tablet server has 3 leaders and 0 scanners
I20250627 13:54:30.802983 8041 tablet_service.cc:1430] Tablet server has 9 leaders and 0 scanners
I20250627 13:54:30.808887 7782 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:30.860261 8171 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:30.861444 9822 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
I20250627 13:54:30.863023 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:30.874784 8171 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:30.876592 7782 tablet_service.cc:1430] Tablet server has 6 leaders and 0 scanners
I20250627 13:54:30.906885 13761 raft_consensus.cc:1062] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096: attempting to promote NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 to VOTER
I20250627 13:54:30.907752 13761 consensus_queue.cc:237] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } }
I20250627 13:54:30.922187 9842 consensus_queue.cc:237] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } }
I20250627 13:54:30.926779 8063 raft_consensus.cc:1273] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Refusing update from remote peer 956ff058e6d64579a1963aabdb6d466b: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250627 13:54:30.929603 9691 tablet_service.cc:1430] Tablet server has 3 leaders and 0 scanners
I20250627 13:54:30.933233 13977 raft_consensus.cc:1062] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096: attempting to promote NON_VOTER 956ff058e6d64579a1963aabdb6d466b to VOTER
I20250627 13:54:30.934352 13977 consensus_queue.cc:237] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } }
I20250627 13:54:30.936683 7933 raft_consensus.cc:1273] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Refusing update from remote peer 956ff058e6d64579a1963aabdb6d466b: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250627 13:54:30.937923 14246 consensus_queue.cc:1035] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250627 13:54:30.941188 9822 tablet_service.cc:1430] Tablet server has 2 leaders and 0 scanners
I20250627 13:54:30.942620 14248 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } }
I20250627 13:54:30.949982 7933 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250627 13:54:30.950965 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 1dc569a7b12d40058d073f7b81b2601c with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:30.950695 14278 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.002s
I20250627 13:54:30.953441 7655 catalog_manager.cc:5582] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b reported cstate change: config changed from index 11 to 13, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New cstate: current_term: 2 leader_uuid: "956ff058e6d64579a1963aabdb6d466b" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:30.956820 8063 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } }
I20250627 13:54:30.957888 10104 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250627 13:54:30.958807 14274 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:30.959134 7782 tablet_service.cc:1515] Processing DeleteTablet for tablet 1dc569a7b12d40058d073f7b81b2601c with delete_type TABLET_DATA_TOMBSTONED (TS 83574c1bf8ea45adaff939158f93c096 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:45826
I20250627 13:54:30.964726 7931 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:30.967271 9708 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:30.967821 14278 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:30.969060 10083 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:30.973913 13977 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:30.973515 14248 consensus_queue.cc:1035] T 1dc569a7b12d40058d073f7b81b2601c P 956ff058e6d64579a1963aabdb6d466b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250627 13:54:30.976141 7931 raft_consensus.cc:2953] T 1dc569a7b12d40058d073f7b81b2601c P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } }
I20250627 13:54:30.987075 14281 tablet_replica.cc:331] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096: stopping tablet replica
I20250627 13:54:30.987303 14281 raft_consensus.cc:2241] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:30.987504 14281 raft_consensus.cc:2270] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:30.994275 14170 raft_consensus.cc:1062] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER e4c831df4da841929d13d6c783099b0e to VOTER
I20250627 13:54:30.995093 14170 consensus_queue.cc:237] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } }
I20250627 13:54:30.997920 14278 raft_consensus.cc:1025] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: attempt to promote peer a41a89ae8ed743edb2b5cf4490d8c854: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250627 13:54:30.998731 14281 ts_tablet_manager.cc:1905] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:31.002072 9840 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250627 13:54:31.006718 14283 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:31.009095 7800 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250627 13:54:31.009727 14279 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250627 13:54:31.011513 8041 tablet_service.cc:1430] Tablet server has 9 leaders and 0 scanners
I20250627 13:54:31.017236 8190 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250627 13:54:31.018532 14277 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250627 13:54:31.018639 10102 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250627 13:54:31.029031 9972 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250627 13:54:31.029493 9973 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250627 13:54:31.029778 14279 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20250627 13:54:31.038558 14274 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250627 13:54:31.037637 14284 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.002s
I20250627 13:54:31.051285 10083 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:31.052736 14170 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:31.054960 14274 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:31.057194 10102 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:31.062288 7661 catalog_manager.cc:5582] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 10 to 11, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:31.063019 7655 catalog_manager.cc:5582] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 reported cstate change: config changed from index 13 to 14, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:31.063625 7800 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:31.064044 7911 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:31.066434 14274 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:31.077062 7927 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:31.080451 10102 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:31.080893 9842 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:31.074203 9708 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:31.091046 7659 catalog_manager.cc:5582] T 897206e017ff4305a8cc9092e30f46b6 P 956ff058e6d64579a1963aabdb6d466b reported cstate change: config changed from index 11 to 12, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:31.097013 13977 raft_consensus.cc:1062] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096: attempting to promote NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 to VOTER
I20250627 13:54:31.097990 13977 consensus_queue.cc:237] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } }
I20250627 13:54:31.100597 7927 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 LEARNER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250627 13:54:31.103575 10102 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250627 13:54:31.103672 9842 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250627 13:54:31.112097 13603 raft_consensus.cc:1062] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a: attempting to promote NON_VOTER e4c831df4da841929d13d6c783099b0e to VOTER
I20250627 13:54:31.112087 14283 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250627 13:54:31.112494 14283 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250627 13:54:31.112795 14283 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250627 13:54:31.112934 13603 consensus_queue.cc:237] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 2.19, Last appended by leader: 16, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } }
I20250627 13:54:31.122109 9973 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250627 13:54:31.122992 7927 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:31.125447 14283 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:31.127308 7926 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:31.142104 14281 ts_tablet_manager.cc:1918] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20250627 13:54:31.142369 14281 log.cc:1199] T 1dc569a7b12d40058d073f7b81b2601c P 83574c1bf8ea45adaff939158f93c096: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/wals/1dc569a7b12d40058d073f7b81b2601c
I20250627 13:54:31.155056 10083 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:31.175882 8189 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:31.178882 8171 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:31.179992 7661 catalog_manager.cc:5582] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index 12 to 13, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:31.185556 8063 consensus_queue.cc:237] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } }
I20250627 13:54:31.188748 8189 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250627 13:54:31.189261 14291 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250627 13:54:31.199714 14291 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:31.204198 8189 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:31.205262 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet d2eb537d49804e1b8ec5a217b1564988 with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:31.206475 7655 catalog_manager.cc:5582] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 14 to 15, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:31.207942 10102 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:31.207945 10104 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250627 13:54:31.212810 14277 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250627 13:54:31.217530 10103 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:31.225625 9974 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 19. Preceding OpId from leader: term: 2 index: 20. (index mismatch)
I20250627 13:54:31.226347 14214 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250627 13:54:31.242482 7931 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 19. Preceding OpId from leader: term: 2 index: 20. (index mismatch)
I20250627 13:54:31.245929 9842 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
W20250627 13:54:31.248030 7738 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.71:37859, user_credentials={real_user=slave}} blocked reactor thread for 124799us
I20250627 13:54:31.256333 9973 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:31.256937 7801 consensus_queue.cc:237] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } }
I20250627 13:54:31.257201 7645 catalog_manager.cc:4928] TS 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467): tablet 1dc569a7b12d40058d073f7b81b2601c (table rebalance_test_table_3 [id=ec6a9ed6ab4d416988b2abe2f8d29c55]) successfully deleted
I20250627 13:54:31.260892 7782 tablet_service.cc:1515] Processing DeleteTablet for tablet d2eb537d49804e1b8ec5a217b1564988 with delete_type TABLET_DATA_TOMBSTONED (TS 83574c1bf8ea45adaff939158f93c096 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:45826
I20250627 13:54:31.260943 7933 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250627 13:54:31.261441 14278 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250627 13:54:31.262116 13603 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250627 13:54:31.262316 14281 tablet_replica.cc:331] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: stopping tablet replica
I20250627 13:54:31.262467 14281 raft_consensus.cc:2241] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:31.262634 14281 raft_consensus.cc:2270] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:31.265550 9840 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250627 13:54:31.266791 14274 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250627 13:54:31.250087 7927 consensus_queue.cc:237] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } }
I20250627 13:54:31.269994 9971 raft_consensus.cc:1273] T 83cfc975906745a09c5e1dcfac812ae2 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250627 13:54:31.270015 14274 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:31.270529 14215 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250627 13:54:31.273088 14215 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:31.275697 7659 catalog_manager.cc:5582] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 13 to 14, VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) evicted. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:31.277608 9971 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:31.277942 9842 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:31.280213 9954 tablet_service.cc:1515] Processing DeleteTablet for tablet 897206e017ff4305a8cc9092e30f46b6 with delete_type TABLET_DATA_TOMBSTONED (TS 6686c6c365bf4f4582058890161676fe not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:35810
I20250627 13:54:31.282918 8190 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 2 LEARNER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 19. Preceding OpId from leader: term: 2 index: 20. (index mismatch)
I20250627 13:54:31.283524 13603 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250627 13:54:31.285091 14281 ts_tablet_manager.cc:1905] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:31.287019 7647 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 83cfc975906745a09c5e1dcfac812ae2 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:31.288098 7931 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 6686c6c365bf4f4582058890161676fe (127.0.103.71) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:31.290911 14214 raft_consensus.cc:1062] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a: attempting to promote NON_VOTER 83574c1bf8ea45adaff939158f93c096 to VOTER
I20250627 13:54:31.291723 14214 consensus_queue.cc:237] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } }
I20250627 13:54:31.294751 7659 catalog_manager.cc:5582] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index 12 to 13, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New cstate: current_term: 2 leader_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:31.296733 10104 raft_consensus.cc:1273] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Refusing update from remote peer a41a89ae8ed743edb2b5cf4490d8c854: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250627 13:54:31.299489 9822 tablet_service.cc:1515] Processing DeleteTablet for tablet 83cfc975906745a09c5e1dcfac812ae2 with delete_type TABLET_DATA_TOMBSTONED (TS 956ff058e6d64579a1963aabdb6d466b not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:55228
I20250627 13:54:31.303560 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 897206e017ff4305a8cc9092e30f46b6 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:31.304746 14215 consensus_queue.cc:1035] T 83cfc975906745a09c5e1dcfac812ae2 P a41a89ae8ed743edb2b5cf4490d8c854 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250627 13:54:31.306624 10100 raft_consensus.cc:2953] T 83cfc975906745a09c5e1dcfac812ae2 P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } }
I20250627 13:54:31.307246 7801 consensus_queue.cc:237] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } }
I20250627 13:54:31.321053 14308 tablet_replica.cc:331] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe: stopping tablet replica
I20250627 13:54:31.321290 14308 raft_consensus.cc:2241] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:31.326071 9840 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250627 13:54:31.327415 14275 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250627 13:54:31.328663 14312 tablet_replica.cc:331] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b: stopping tablet replica
I20250627 13:54:31.328938 14312 raft_consensus.cc:2241] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:31.329275 14312 raft_consensus.cc:2270] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:31.330551 14312 ts_tablet_manager.cc:1905] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:31.330974 13761 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:31.331615 9840 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:31.334586 7801 raft_consensus.cc:1273] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 2 LEARNER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250627 13:54:31.334848 9971 raft_consensus.cc:1273] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250627 13:54:31.335391 14232 consensus_queue.cc:1035] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:31.335939 14232 consensus_queue.cc:1035] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:31.337503 7926 raft_consensus.cc:1273] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250627 13:54:31.337975 14281 ts_tablet_manager.cc:1918] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.14
I20250627 13:54:31.338249 13603 consensus_queue.cc:1035] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:31.347256 14232 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 LEADER]: Committing config change with OpId 2.20: config changed from index 19 to 20, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:31.351675 7927 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250627 13:54:31.353752 14281 log.cc:1199] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/wals/d2eb537d49804e1b8ec5a217b1564988
I20250627 13:54:31.352442 14275 consensus_queue.cc:1035] T 897206e017ff4305a8cc9092e30f46b6 P 83574c1bf8ea45adaff939158f93c096 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250627 13:54:31.356612 7927 raft_consensus.cc:2953] T 897206e017ff4305a8cc9092e30f46b6 P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:31.357771 7655 catalog_manager.cc:5582] T 897206e017ff4305a8cc9092e30f46b6 P 956ff058e6d64579a1963aabdb6d466b reported cstate change: config changed from index 14 to 15, VOTER dede81a8c0164427a61532d01b6fc157 (127.0.103.72) evicted. New cstate: current_term: 1 leader_uuid: "83574c1bf8ea45adaff939158f93c096" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:31.361550 7645 catalog_manager.cc:4928] TS 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467): tablet d2eb537d49804e1b8ec5a217b1564988 (table rebalance_test_table_0 [id=d8bfc9268a4a4c2c92bc20d56ac73086]) successfully deleted
I20250627 13:54:31.362790 10083 tablet_service.cc:1515] Processing DeleteTablet for tablet 897206e017ff4305a8cc9092e30f46b6 with delete_type TABLET_DATA_TOMBSTONED (TS dede81a8c0164427a61532d01b6fc157 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:56490
I20250627 13:54:31.364675 14308 pending_rounds.cc:70] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe: Trying to abort 1 pending ops.
I20250627 13:54:31.364907 14308 pending_rounds.cc:77] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe: Aborting op as it isn't in flight: id { term: 1 index: 12 } timestamp: 7172229000944119808 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "897206e017ff4305a8cc9092e30f46b6" old_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } } new_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } } }
I20250627 13:54:31.365504 14308 raft_consensus.cc:2915] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Aborting config change with OpId 1.12: Aborted: Op aborted
I20250627 13:54:31.365718 14308 raft_consensus.cc:2887] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250627 13:54:31.365888 14308 raft_consensus.cc:2270] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:31.366855 14308 ts_tablet_manager.cc:1905] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:31.374500 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 897206e017ff4305a8cc9092e30f46b6 with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:31.379580 8189 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Committing config change with OpId 2.20: config changed from index 19 to 20, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:31.378182 9974 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Committing config change with OpId 2.20: config changed from index 19 to 20, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:31.398175 7931 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Committing config change with OpId 2.20: config changed from index 19 to 20, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:31.402485 7661 catalog_manager.cc:5582] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: config changed from index 19 to 20, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:31.417129 10103 raft_consensus.cc:1273] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 83574c1bf8ea45adaff939158f93c096: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250627 13:54:31.417574 9709 consensus_queue.cc:237] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 2.20, Last appended by leader: 16, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } }
I20250627 13:54:31.422665 13603 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:31.425100 8189 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 20. Preceding OpId from leader: term: 2 index: 21. (index mismatch)
I20250627 13:54:31.425323 14318 tablet_replica.cc:331] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157: stopping tablet replica
I20250627 13:54:31.425572 14318 raft_consensus.cc:2241] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:31.426018 14318 raft_consensus.cc:2270] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:31.426898 14232 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250627 13:54:31.428615 9972 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:31.429133 14308 ts_tablet_manager.cc:1918] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20250627 13:54:31.429252 14308 log.cc:1199] T 897206e017ff4305a8cc9092e30f46b6 P 6686c6c365bf4f4582058890161676fe: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-6/wal/wals/897206e017ff4305a8cc9092e30f46b6
I20250627 13:54:31.429603 9973 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 20. Preceding OpId from leader: term: 2 index: 21. (index mismatch)
I20250627 13:54:31.430209 7648 catalog_manager.cc:4928] TS 6686c6c365bf4f4582058890161676fe (127.0.103.71:37859): tablet 897206e017ff4305a8cc9092e30f46b6 (table rebalance_test_table_4 [id=4d92d465970b4ec4b5eace263b25362a]) successfully deleted
I20250627 13:54:31.430346 14232 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250627 13:54:31.430105 7927 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:31.434114 14214 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 LEADER]: Committing config change with OpId 2.21: config changed from index 20 to 21, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:31.437163 9973 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Committing config change with OpId 2.21: config changed from index 20 to 21, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:31.439028 14318 ts_tablet_manager.cc:1905] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:31.439299 7659 catalog_manager.cc:5582] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe reported cstate change: config changed from index 11 to 12, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:31.447386 7659 catalog_manager.cc:5582] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe reported cstate change: config changed from index 20 to 21, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New cstate: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:31.451171 7911 tablet_service.cc:1515] Processing DeleteTablet for tablet e3053b529a264e8697a1df9e17eeea81 with delete_type TABLET_DATA_TOMBSTONED (TS a41a89ae8ed743edb2b5cf4490d8c854 not found in new config with opid_index 21) from {username='slave'} at 127.0.0.1:59302
I20250627 13:54:31.451632 8189 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Committing config change with OpId 2.21: config changed from index 20 to 21, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:31.458957 8063 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5184c3f8e465472e9a98c14f8e703415"
dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35046
I20250627 13:54:31.459228 8063 raft_consensus.cc:604] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Received request to transfer leadership
I20250627 13:54:31.469043 14312 ts_tablet_manager.cc:1918] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20250627 13:54:31.470468 14323 tablet_replica.cc:331] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854: stopping tablet replica
I20250627 13:54:31.470932 14323 raft_consensus.cc:2241] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:31.471187 14323 raft_consensus.cc:2270] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:31.472146 14323 ts_tablet_manager.cc:1905] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:31.479601 7648 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet e3053b529a264e8697a1df9e17eeea81 with cas_config_opid_index 20: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:31.490888 7801 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { replace: true } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:31.500437 9709 consensus_queue.cc:237] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } }
I20250627 13:54:31.504140 9973 raft_consensus.cc:1273] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250627 13:54:31.505101 14321 consensus_queue.cc:1035] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250627 13:54:31.508350 14232 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:31.510536 14312 log.cc:1199] T 83cfc975906745a09c5e1dcfac812ae2 P 956ff058e6d64579a1963aabdb6d466b: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal/wals/83cfc975906745a09c5e1dcfac812ae2
I20250627 13:54:31.511420 7645 catalog_manager.cc:4928] TS 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): tablet 83cfc975906745a09c5e1dcfac812ae2 (table rebalance_test_table_9 [id=19e5686b50fe4187a733aa6413ef2ce0]) successfully deleted
I20250627 13:54:31.516683 9973 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:31.524750 7648 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 40431e3bd7244904a4e65f59477bd21d with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:31.525882 14318 ts_tablet_manager.cc:1918] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.13
I20250627 13:54:31.526252 14318 log.cc:1199] T 897206e017ff4305a8cc9092e30f46b6 P dede81a8c0164427a61532d01b6fc157: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-7/wal/wals/897206e017ff4305a8cc9092e30f46b6
I20250627 13:54:31.526418 7659 catalog_manager.cc:5582] T 40431e3bd7244904a4e65f59477bd21d P 6686c6c365bf4f4582058890161676fe reported cstate change: config changed from index 12 to 13, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New cstate: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:31.527249 7647 catalog_manager.cc:4928] TS dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213): tablet 897206e017ff4305a8cc9092e30f46b6 (table rebalance_test_table_4 [id=4d92d465970b4ec4b5eace263b25362a]) successfully deleted
I20250627 13:54:31.531212 7911 tablet_service.cc:1515] Processing DeleteTablet for tablet 40431e3bd7244904a4e65f59477bd21d with delete_type TABLET_DATA_TOMBSTONED (TS a41a89ae8ed743edb2b5cf4490d8c854 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:59302
I20250627 13:54:31.532989 7800 raft_consensus.cc:1273] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250627 13:54:31.534909 14232 consensus_queue.cc:1035] T 40431e3bd7244904a4e65f59477bd21d P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250627 13:54:31.537210 7800 raft_consensus.cc:2953] T 40431e3bd7244904a4e65f59477bd21d P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:31.547508 14323 ts_tablet_manager.cc:1918] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.20
I20250627 13:54:31.548662 14323 log.cc:1199] T e3053b529a264e8697a1df9e17eeea81 P a41a89ae8ed743edb2b5cf4490d8c854: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/wals/e3053b529a264e8697a1df9e17eeea81
I20250627 13:54:31.549607 7647 catalog_manager.cc:4928] TS a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675): tablet e3053b529a264e8697a1df9e17eeea81 (table rebalance_test_table_0 [id=d8bfc9268a4a4c2c92bc20d56ac73086]) successfully deleted
I20250627 13:54:31.550113 14323 tablet_replica.cc:331] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854: stopping tablet replica
I20250627 13:54:31.550354 14323 raft_consensus.cc:2241] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:31.550585 14323 raft_consensus.cc:2270] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:31.555481 14323 ts_tablet_manager.cc:1905] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:31.581444 14323 ts_tablet_manager.cc:1918] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20250627 13:54:31.583600 14323 log.cc:1199] T 40431e3bd7244904a4e65f59477bd21d P a41a89ae8ed743edb2b5cf4490d8c854: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/wals/40431e3bd7244904a4e65f59477bd21d
I20250627 13:54:31.602789 7647 catalog_manager.cc:4928] TS a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675): tablet 40431e3bd7244904a4e65f59477bd21d (table rebalance_test_table_2 [id=6defd4a16efc47569b2193b71e4b2c4a]) successfully deleted
I20250627 13:54:31.678766 8189 consensus_queue.cc:237] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: NON_VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: true } }
I20250627 13:54:31.681528 9842 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20250627 13:54:31.682219 8126 consensus_peers.cc:487] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e -> Peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933): Couldn't send request to peer 9cf479bdcd0242738d2a77db4ad9f660. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14dba72d45694defb83f41c1572722ca. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:31.682514 13848 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:31.685096 10103 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250627 13:54:31.685750 13848 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:31.687583 14277 raft_consensus.cc:991] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660: : Instructing follower 956ff058e6d64579a1963aabdb6d466b to start an election
I20250627 13:54:31.687678 14277 raft_consensus.cc:1079] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Signalling peer 956ff058e6d64579a1963aabdb6d466b to start an election
I20250627 13:54:31.688412 9840 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "5184c3f8e465472e9a98c14f8e703415"
dest_uuid: "956ff058e6d64579a1963aabdb6d466b"
from {username='slave'} at 127.0.103.67:41969
I20250627 13:54:31.688611 9840 raft_consensus.cc:491] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:54:31.688710 9840 raft_consensus.cc:3058] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:31.692931 9840 raft_consensus.cc:513] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:31.694262 8063 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5184c3f8e465472e9a98c14f8e703415" candidate_uuid: "956ff058e6d64579a1963aabdb6d466b" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660"
I20250627 13:54:31.694495 8063 raft_consensus.cc:3053] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Stepping down as leader of term 1
I20250627 13:54:31.694568 8063 raft_consensus.cc:738] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Running, Role: LEADER
I20250627 13:54:31.694867 8063 consensus_queue.cc:260] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:31.695310 8063 raft_consensus.cc:3058] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:31.700745 8063 raft_consensus.cc:2466] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 956ff058e6d64579a1963aabdb6d466b in term 2.
I20250627 13:54:31.702034 7801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5184c3f8e465472e9a98c14f8e703415" candidate_uuid: "956ff058e6d64579a1963aabdb6d466b" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "83574c1bf8ea45adaff939158f93c096"
I20250627 13:54:31.702347 7801 raft_consensus.cc:3058] T 5184c3f8e465472e9a98c14f8e703415 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:31.705631 7801 raft_consensus.cc:2466] T 5184c3f8e465472e9a98c14f8e703415 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 956ff058e6d64579a1963aabdb6d466b in term 2.
I20250627 13:54:31.707511 9840 leader_election.cc:290] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b [CANDIDATE]: Term 2 election: Requested vote from peers 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933), 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467), 6686c6c365bf4f4582058890161676fe (127.0.103.71:37859)
I20250627 13:54:31.709239 9775 leader_election.cc:304] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 83574c1bf8ea45adaff939158f93c096, 956ff058e6d64579a1963aabdb6d466b, 9cf479bdcd0242738d2a77db4ad9f660; no voters:
I20250627 13:54:31.709551 14324 raft_consensus.cc:2802] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:54:31.709739 14324 raft_consensus.cc:695] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b [term 2 LEADER]: Becoming Leader. State: Replica: 956ff058e6d64579a1963aabdb6d466b, State: Running, Role: LEADER
I20250627 13:54:31.710050 14324 consensus_queue.cc:237] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:31.733390 14362 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: NON_VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: true } } }
I20250627 13:54:31.743418 9842 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: NON_VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: true } } }
I20250627 13:54:31.752627 7655 catalog_manager.cc:5582] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b reported cstate change: term changed from 1 to 2, leader changed from 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) to 956ff058e6d64579a1963aabdb6d466b (127.0.103.70). New cstate: current_term: 2 leader_uuid: "956ff058e6d64579a1963aabdb6d466b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:31.759707 7655 catalog_manager.cc:5582] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b reported cstate change: config changed from index 11 to 12, NON_VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) added. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: NON_VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: true } } }
I20250627 13:54:31.771168 10103 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: NON_VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: true } } }
I20250627 13:54:31.792366 7801 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "87726824b05a4b968386f57d3a7a4aad"
dest_uuid: "83574c1bf8ea45adaff939158f93c096"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59084
I20250627 13:54:31.792588 7801 raft_consensus.cc:604] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Received request to transfer leadership
I20250627 13:54:31.795023 9973 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5184c3f8e465472e9a98c14f8e703415" candidate_uuid: "956ff058e6d64579a1963aabdb6d466b" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "6686c6c365bf4f4582058890161676fe"
I20250627 13:54:31.795269 9973 raft_consensus.cc:3058] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:31.798300 9973 raft_consensus.cc:2466] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 956ff058e6d64579a1963aabdb6d466b in term 2.
I20250627 13:54:31.812021 8063 consensus_queue.cc:237] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 1.18, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } }
W20250627 13:54:31.844471 7996 consensus_peers.cc:487] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467): Couldn't send request to peer 83574c1bf8ea45adaff939158f93c096. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:31.846131 7927 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch)
I20250627 13:54:31.847855 14170 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250627 13:54:31.859881 8189 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch)
I20250627 13:54:31.860430 14170 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250627 13:54:31.891814 14277 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.19: config changed from index 18 to 19, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } }
I20250627 13:54:31.892596 8190 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.19: config changed from index 18 to 19, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } }
I20250627 13:54:31.899356 7661 catalog_manager.cc:5582] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 18 to 19, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } }
I20250627 13:54:31.910645 7931 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.19: config changed from index 18 to 19, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } }
I20250627 13:54:31.953553 14275 raft_consensus.cc:991] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096: : Instructing follower 2a1282ebe92a4c9a92eed3b13f69012a to start an election
I20250627 13:54:31.953737 14275 raft_consensus.cc:1079] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Signalling peer 2a1282ebe92a4c9a92eed3b13f69012a to start an election
I20250627 13:54:31.954576 9709 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "87726824b05a4b968386f57d3a7a4aad"
dest_uuid: "2a1282ebe92a4c9a92eed3b13f69012a"
from {username='slave'} at 127.0.103.65:56847
I20250627 13:54:31.954843 9709 raft_consensus.cc:491] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:54:31.954982 9709 raft_consensus.cc:3058] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:31.960278 9709 raft_consensus.cc:513] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } }
I20250627 13:54:31.961565 7801 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87726824b05a4b968386f57d3a7a4aad" candidate_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "83574c1bf8ea45adaff939158f93c096"
I20250627 13:54:31.961799 7801 raft_consensus.cc:3053] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Stepping down as leader of term 1
I20250627 13:54:31.961884 7801 raft_consensus.cc:738] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Running, Role: LEADER
I20250627 13:54:31.962204 7801 consensus_queue.cc:260] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } }
I20250627 13:54:31.962678 7801 raft_consensus.cc:3058] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:31.966487 7801 raft_consensus.cc:2466] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1282ebe92a4c9a92eed3b13f69012a in term 2.
I20250627 13:54:31.968768 9973 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87726824b05a4b968386f57d3a7a4aad" candidate_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "6686c6c365bf4f4582058890161676fe"
I20250627 13:54:31.968991 9973 raft_consensus.cc:3058] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:31.972494 9973 raft_consensus.cc:2466] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1282ebe92a4c9a92eed3b13f69012a in term 2.
I20250627 13:54:31.972944 9647 leader_election.cc:304] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2a1282ebe92a4c9a92eed3b13f69012a, 6686c6c365bf4f4582058890161676fe, 83574c1bf8ea45adaff939158f93c096; no voters:
I20250627 13:54:31.973392 9709 leader_election.cc:290] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [CANDIDATE]: Term 2 election: Requested vote from peers 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467), 6686c6c365bf4f4582058890161676fe (127.0.103.71:37859), a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675)
I20250627 13:54:31.973848 14319 raft_consensus.cc:2802] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:54:31.974025 14319 raft_consensus.cc:695] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 LEADER]: Becoming Leader. State: Replica: 2a1282ebe92a4c9a92eed3b13f69012a, State: Running, Role: LEADER
I20250627 13:54:31.974468 14319 consensus_queue.cc:237] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } }
I20250627 13:54:31.976454 7927 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87726824b05a4b968386f57d3a7a4aad" candidate_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "a41a89ae8ed743edb2b5cf4490d8c854"
I20250627 13:54:31.976626 7927 raft_consensus.cc:3058] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:31.978480 7659 catalog_manager.cc:5582] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: term changed from 1 to 2, leader changed from 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) to 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69). New cstate: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:31.979350 7927 raft_consensus.cc:2466] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1282ebe92a4c9a92eed3b13f69012a in term 2.
I20250627 13:54:32.060685 14477 ts_tablet_manager.cc:927] T 14dba72d45694defb83f41c1572722ca P 9cf479bdcd0242738d2a77db4ad9f660: Initiating tablet copy from peer e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:54:32.061237 14477 tablet_copy_client.cc:323] T 14dba72d45694defb83f41c1572722ca P 9cf479bdcd0242738d2a77db4ad9f660: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.68:46139
I20250627 13:54:32.061914 8213 tablet_copy_service.cc:140] P e4c831df4da841929d13d6c783099b0e: Received BeginTabletCopySession request for tablet 14dba72d45694defb83f41c1572722ca from peer 9cf479bdcd0242738d2a77db4ad9f660 ({username='slave'} at 127.0.103.67:37199)
I20250627 13:54:32.062057 8213 tablet_copy_service.cc:161] P e4c831df4da841929d13d6c783099b0e: Beginning new tablet copy session on tablet 14dba72d45694defb83f41c1572722ca from peer 9cf479bdcd0242738d2a77db4ad9f660 at {username='slave'} at 127.0.103.67:37199: session id = 9cf479bdcd0242738d2a77db4ad9f660-14dba72d45694defb83f41c1572722ca
I20250627 13:54:32.064102 8213 tablet_copy_source_session.cc:215] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:32.065412 14477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14dba72d45694defb83f41c1572722ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:32.075157 14477 tablet_copy_client.cc:806] T 14dba72d45694defb83f41c1572722ca P 9cf479bdcd0242738d2a77db4ad9f660: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:32.075377 14477 tablet_copy_client.cc:670] T 14dba72d45694defb83f41c1572722ca P 9cf479bdcd0242738d2a77db4ad9f660: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:32.078172 14477 tablet_copy_client.cc:538] T 14dba72d45694defb83f41c1572722ca P 9cf479bdcd0242738d2a77db4ad9f660: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:32.086392 14477 tablet_bootstrap.cc:492] T 14dba72d45694defb83f41c1572722ca P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap starting.
I20250627 13:54:32.150731 7801 raft_consensus.cc:1273] T 5184c3f8e465472e9a98c14f8e703415 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Refusing update from remote peer 956ff058e6d64579a1963aabdb6d466b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250627 13:54:32.151453 14324 consensus_queue.cc:1035] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:32.166123 14477 tablet_bootstrap.cc:492] T 14dba72d45694defb83f41c1572722ca P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:32.166581 14477 tablet_bootstrap.cc:492] T 14dba72d45694defb83f41c1572722ca P 9cf479bdcd0242738d2a77db4ad9f660: Bootstrap complete.
I20250627 13:54:32.166870 14477 ts_tablet_manager.cc:1397] T 14dba72d45694defb83f41c1572722ca P 9cf479bdcd0242738d2a77db4ad9f660: Time spent bootstrapping tablet: real 0.081s user 0.058s sys 0.010s
I20250627 13:54:32.167762 14477 raft_consensus.cc:357] T 14dba72d45694defb83f41c1572722ca P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: NON_VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: true } }
I20250627 13:54:32.168030 14477 raft_consensus.cc:738] T 14dba72d45694defb83f41c1572722ca P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Initialized, Role: LEARNER
I20250627 13:54:32.168251 14477 consensus_queue.cc:260] T 14dba72d45694defb83f41c1572722ca P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: NON_VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: true } }
I20250627 13:54:32.168928 14477 ts_tablet_manager.cc:1428] T 14dba72d45694defb83f41c1572722ca P 9cf479bdcd0242738d2a77db4ad9f660: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250627 13:54:32.169719 8213 tablet_copy_service.cc:342] P e4c831df4da841929d13d6c783099b0e: Request end of tablet copy session 9cf479bdcd0242738d2a77db4ad9f660-14dba72d45694defb83f41c1572722ca received from {username='slave'} at 127.0.103.67:37199
I20250627 13:54:32.169858 8213 tablet_copy_service.cc:434] P e4c831df4da841929d13d6c783099b0e: ending tablet copy session 9cf479bdcd0242738d2a77db4ad9f660-14dba72d45694defb83f41c1572722ca on tablet 14dba72d45694defb83f41c1572722ca with peer 9cf479bdcd0242738d2a77db4ad9f660
I20250627 13:54:32.186149 9973 raft_consensus.cc:1273] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Refusing update from remote peer 956ff058e6d64579a1963aabdb6d466b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250627 13:54:32.186880 14324 consensus_queue.cc:1035] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:32.193531 8063 raft_consensus.cc:1273] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Refusing update from remote peer 956ff058e6d64579a1963aabdb6d466b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250627 13:54:32.194273 14324 consensus_queue.cc:1035] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250627 13:54:32.242379 9842 consensus_queue.cc:237] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:32.247833 9973 raft_consensus.cc:1273] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Refusing update from remote peer 956ff058e6d64579a1963aabdb6d466b: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250627 13:54:32.248667 14324 consensus_queue.cc:1035] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:32.251987 14324 raft_consensus.cc:2953] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:32.254272 9974 raft_consensus.cc:2953] T 5184c3f8e465472e9a98c14f8e703415 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:32.260627 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 5184c3f8e465472e9a98c14f8e703415 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:32.261715 7659 catalog_manager.cc:5582] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b reported cstate change: config changed from index 8 to 10, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New cstate: current_term: 2 leader_uuid: "956ff058e6d64579a1963aabdb6d466b" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:32.263437 7800 raft_consensus.cc:1273] T 5184c3f8e465472e9a98c14f8e703415 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Refusing update from remote peer 956ff058e6d64579a1963aabdb6d466b: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250627 13:54:32.267081 14324 consensus_queue.cc:1035] T 5184c3f8e465472e9a98c14f8e703415 P 956ff058e6d64579a1963aabdb6d466b [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250627 13:54:32.267951 8041 tablet_service.cc:1515] Processing DeleteTablet for tablet 5184c3f8e465472e9a98c14f8e703415 with delete_type TABLET_DATA_TOMBSTONED (TS 9cf479bdcd0242738d2a77db4ad9f660 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:59760
I20250627 13:54:32.269565 7801 raft_consensus.cc:2953] T 5184c3f8e465472e9a98c14f8e703415 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:32.278817 14527 tablet_replica.cc:331] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660: stopping tablet replica
I20250627 13:54:32.279116 14527 raft_consensus.cc:2241] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:32.279403 14527 raft_consensus.cc:2270] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:32.280474 14527 ts_tablet_manager.cc:1905] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:32.286033 14527 ts_tablet_manager.cc:1918] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20250627 13:54:32.286201 14527 log.cc:1199] T 5184c3f8e465472e9a98c14f8e703415 P 9cf479bdcd0242738d2a77db4ad9f660: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-2/wal/wals/5184c3f8e465472e9a98c14f8e703415
I20250627 13:54:32.286979 7645 catalog_manager.cc:4928] TS 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933): tablet 5184c3f8e465472e9a98c14f8e703415 (table rebalance_test_table_3 [id=ec6a9ed6ab4d416988b2abe2f8d29c55]) successfully deleted
I20250627 13:54:32.308983 14534 ts_tablet_manager.cc:927] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:32.315811 14534 tablet_copy_client.cc:323] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:32.316421 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet 644985e3086d4614b6598a727cb60c1d from peer 83574c1bf8ea45adaff939158f93c096 ({username='slave'} at 127.0.103.65:50643)
I20250627 13:54:32.316552 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet 644985e3086d4614b6598a727cb60c1d from peer 83574c1bf8ea45adaff939158f93c096 at {username='slave'} at 127.0.103.65:50643: session id = 83574c1bf8ea45adaff939158f93c096-644985e3086d4614b6598a727cb60c1d
I20250627 13:54:32.319479 8082 tablet_copy_source_session.cc:215] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:32.320586 14534 ts_tablet_manager.cc:1905] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250627 13:54:32.327283 14534 ts_tablet_manager.cc:1918] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.17
I20250627 13:54:32.327605 14534 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 644985e3086d4614b6598a727cb60c1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:32.329756 14534 tablet_copy_client.cc:806] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:32.329936 14534 tablet_copy_client.cc:670] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:32.331771 14534 tablet_copy_client.cc:538] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:32.334224 14534 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:54:32.366602 7927 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250627 13:54:32.367349 14321 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:32.374825 9691 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:32.379664 9691 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:32.418761 8041 tablet_service.cc:1430] Tablet server has 8 leaders and 0 scanners
I20250627 13:54:32.424424 7782 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:32.431360 7911 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:32.448625 7801 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250627 13:54:32.449397 14321 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:32.466922 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:32.501134 9974 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250627 13:54:32.501818 9822 tablet_service.cc:1430] Tablet server has 3 leaders and 0 scanners
I20250627 13:54:32.502965 14321 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250627 13:54:32.513410 10083 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:32.530638 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:32.544250 7782 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:32.574407 7911 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:32.575258 9822 tablet_service.cc:1430] Tablet server has 3 leaders and 0 scanners
I20250627 13:54:32.597858 10083 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:32.605821 9709 consensus_queue.cc:237] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } }
I20250627 13:54:32.618819 8041 tablet_service.cc:1430] Tablet server has 8 leaders and 0 scanners
I20250627 13:54:32.620960 7931 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250627 13:54:32.622087 14321 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250627 13:54:32.625339 14321 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:32.629614 7648 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 87726824b05a4b968386f57d3a7a4aad with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:32.632439 7659 catalog_manager.cc:5582] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: config changed from index 11 to 13, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New cstate: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:32.623493 9974 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250627 13:54:32.634887 7927 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:32.635212 14568 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250627 13:54:32.636660 7782 tablet_service.cc:1515] Processing DeleteTablet for tablet 87726824b05a4b968386f57d3a7a4aad with delete_type TABLET_DATA_TOMBSTONED (TS 83574c1bf8ea45adaff939158f93c096 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:45826
I20250627 13:54:32.667172 9974 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:32.724400 14580 tablet_replica.cc:331] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096: stopping tablet replica
I20250627 13:54:32.724679 14580 raft_consensus.cc:2241] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:32.724874 14580 raft_consensus.cc:2270] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:32.730962 14580 ts_tablet_manager.cc:1905] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:32.748764 14534 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096: Bootstrap replayed 1/1 log segments. Stats: ops{read=19 overwritten=0 applied=19 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:32.749205 14534 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096: Bootstrap complete.
I20250627 13:54:32.749450 14534 ts_tablet_manager.cc:1397] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.415s user 0.104s sys 0.094s
I20250627 13:54:32.753336 14580 ts_tablet_manager.cc:1918] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20250627 13:54:32.753448 14580 log.cc:1199] T 87726824b05a4b968386f57d3a7a4aad P 83574c1bf8ea45adaff939158f93c096: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/wals/87726824b05a4b968386f57d3a7a4aad
I20250627 13:54:32.754710 7645 catalog_manager.cc:4928] TS 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467): tablet 87726824b05a4b968386f57d3a7a4aad (table rebalance_test_table_1 [id=8961aa0df2f1488f82cd55cfa22ad214]) successfully deleted
I20250627 13:54:32.755849 14534 raft_consensus.cc:357] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } }
I20250627 13:54:32.756083 14534 raft_consensus.cc:738] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: LEARNER
I20250627 13:54:32.756283 14534 consensus_queue.cc:260] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 19, Last appended: 1.19, Last appended by leader: 19, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } }
I20250627 13:54:32.756762 14534 ts_tablet_manager.cc:1428] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250627 13:54:32.758224 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session 83574c1bf8ea45adaff939158f93c096-644985e3086d4614b6598a727cb60c1d received from {username='slave'} at 127.0.103.65:50643
I20250627 13:54:32.758343 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session 83574c1bf8ea45adaff939158f93c096-644985e3086d4614b6598a727cb60c1d on tablet 644985e3086d4614b6598a727cb60c1d with peer 83574c1bf8ea45adaff939158f93c096
I20250627 13:54:32.765878 8171 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:32.766150 8063 raft_consensus.cc:1215] T 14dba72d45694defb83f41c1572722ca P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20250627 13:54:32.834110 8171 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:32.884459 7801 raft_consensus.cc:1215] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.18->[1.19-1.19] Dedup: 1.19->[]
I20250627 13:54:32.991978 8063 consensus_queue.cc:237] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } }
W20250627 13:54:32.998831 7996 consensus_peers.cc:487] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467): Couldn't send request to peer 83574c1bf8ea45adaff939158f93c096. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:32.998847 8189 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20250627 13:54:32.999023 10102 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20250627 13:54:32.999390 14597 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250627 13:54:32.999601 14585 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250627 13:54:33.004233 14597 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } }
I20250627 13:54:33.005362 10102 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } }
I20250627 13:54:33.009905 8189 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } }
I20250627 13:54:33.010164 7661 catalog_manager.cc:5582] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 15 to 16, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:33.033905 9709 consensus_queue.cc:237] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 2.21, Last appended by leader: 16, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } }
I20250627 13:54:33.034754 8063 consensus_queue.cc:237] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } }
W20250627 13:54:33.036741 9644 consensus_peers.cc:487] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a -> Peer 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467): Couldn't send request to peer 83574c1bf8ea45adaff939158f93c096. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:33.036744 9974 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 21. Preceding OpId from leader: term: 2 index: 22. (index mismatch)
I20250627 13:54:33.037273 8189 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 21. Preceding OpId from leader: term: 2 index: 22. (index mismatch)
I20250627 13:54:33.037393 8190 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250627 13:54:33.037745 9709 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250627 13:54:33.038264 14624 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20250627 13:54:33.038543 7996 consensus_peers.cc:487] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467): Couldn't send request to peer 83574c1bf8ea45adaff939158f93c096. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:33.038516 14321 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250627 13:54:33.038744 14624 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250627 13:54:33.038841 14564 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250627 13:54:33.041240 14624 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } }
I20250627 13:54:33.044986 9709 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } }
I20250627 13:54:33.046028 8189 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } }
I20250627 13:54:33.048091 7653 catalog_manager.cc:5582] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 12 to 13, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:33.050854 14568 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 LEADER]: Committing config change with OpId 2.22: config changed from index 21 to 22, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } }
I20250627 13:54:33.056499 9974 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Committing config change with OpId 2.22: config changed from index 21 to 22, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } }
I20250627 13:54:33.057734 8189 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Committing config change with OpId 2.22: config changed from index 21 to 22, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } }
I20250627 13:54:33.066649 7653 catalog_manager.cc:5582] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 21 to 22, NON_VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) added. New cstate: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } }
I20250627 13:54:33.093245 8063 consensus_queue.cc:237] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
W20250627 13:54:33.100895 7996 consensus_peers.cc:487] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:33.101951 8189 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250627 13:54:33.103036 9709 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250627 13:54:33.103307 14624 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250627 13:54:33.103726 14624 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20250627 13:54:33.105096 7996 consensus_peers.cc:487] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467): Couldn't send request to peer 83574c1bf8ea45adaff939158f93c096. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:33.106504 14624 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:33.109213 8189 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:33.113631 9709 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:33.117748 7653 catalog_manager.cc:5582] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 13 to 14, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:33.229243 9691 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:33.256479 14362 raft_consensus.cc:1062] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e: attempting to promote NON_VOTER 9cf479bdcd0242738d2a77db4ad9f660 to VOTER
I20250627 13:54:33.257390 14362 consensus_queue.cc:237] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } }
I20250627 13:54:33.270058 8062 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEARNER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250627 13:54:33.270560 14582 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250627 13:54:33.273566 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:33.277590 9842 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250627 13:54:33.280663 14582 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250627 13:54:33.284660 9822 tablet_service.cc:1430] Tablet server has 3 leaders and 0 scanners
I20250627 13:54:33.291054 10102 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250627 13:54:33.291671 14582 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250627 13:54:33.300109 14582 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } }
I20250627 13:54:33.303568 8062 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } }
I20250627 13:54:33.313097 9842 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } }
I20250627 13:54:33.317530 7659 catalog_manager.cc:5582] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 12 to 13, 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:33.327916 7782 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:33.334357 8171 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:33.342147 10102 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } }
I20250627 13:54:33.349545 10083 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:33.352800 8189 consensus_queue.cc:237] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } }
I20250627 13:54:33.358855 8062 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250627 13:54:33.359477 14582 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250627 13:54:33.368176 10102 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250627 13:54:33.368676 8041 tablet_service.cc:1430] Tablet server has 8 leaders and 0 scanners
I20250627 13:54:33.368750 14714 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.002s
I20250627 13:54:33.371990 14582 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } }
I20250627 13:54:33.373391 8063 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } }
I20250627 13:54:33.380524 7911 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:33.390777 7652 catalog_manager.cc:5582] T 14dba72d45694defb83f41c1572722ca P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 13 to 14, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } }
I20250627 13:54:33.395364 9822 tablet_service.cc:1515] Processing DeleteTablet for tablet 14dba72d45694defb83f41c1572722ca with delete_type TABLET_DATA_TOMBSTONED (TS 956ff058e6d64579a1963aabdb6d466b not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:55228
I20250627 13:54:33.397352 10103 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } }
I20250627 13:54:33.408294 7648 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 14dba72d45694defb83f41c1572722ca with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:33.421090 14718 tablet_replica.cc:331] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: stopping tablet replica
I20250627 13:54:33.421456 14718 raft_consensus.cc:2241] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:33.426013 14718 raft_consensus.cc:2270] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:33.427362 14718 ts_tablet_manager.cc:1905] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:33.447566 14720 ts_tablet_manager.cc:927] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:33.471131 14720 tablet_copy_client.cc:323] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:33.471832 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet d2eb537d49804e1b8ec5a217b1564988 from peer 83574c1bf8ea45adaff939158f93c096 ({username='slave'} at 127.0.103.65:50643)
I20250627 13:54:33.471937 14718 ts_tablet_manager.cc:1918] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.13
I20250627 13:54:33.471966 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet d2eb537d49804e1b8ec5a217b1564988 from peer 83574c1bf8ea45adaff939158f93c096 at {username='slave'} at 127.0.103.65:50643: session id = 83574c1bf8ea45adaff939158f93c096-d2eb537d49804e1b8ec5a217b1564988
I20250627 13:54:33.472116 14718 log.cc:1199] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal/wals/14dba72d45694defb83f41c1572722ca
I20250627 13:54:33.473088 7645 catalog_manager.cc:4928] TS 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): tablet 14dba72d45694defb83f41c1572722ca (table rebalance_test_table_6 [id=f83cab74056a4e7681a701169d1c1716]) successfully deleted
I20250627 13:54:33.474942 8082 tablet_copy_source_session.cc:215] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:33.475889 14720 ts_tablet_manager.cc:1905] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250627 13:54:33.482733 14624 raft_consensus.cc:1062] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER 83574c1bf8ea45adaff939158f93c096 to VOTER
I20250627 13:54:33.483517 14624 consensus_queue.cc:237] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 1.19, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } }
I20250627 13:54:33.496733 14720 ts_tablet_manager.cc:1918] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.14
I20250627 13:54:33.497054 14720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2eb537d49804e1b8ec5a217b1564988. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:33.499300 7800 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch)
I20250627 13:54:33.499977 14597 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250627 13:54:33.514226 14720 tablet_copy_client.cc:806] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:33.514530 14720 tablet_copy_client.cc:670] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:33.516697 14720 tablet_copy_client.cc:538] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:33.528070 14720 tablet_bootstrap.cc:492] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:54:33.529273 8190 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch)
I20250627 13:54:33.529886 14597 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250627 13:54:33.573887 7927 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch)
I20250627 13:54:33.574376 14624 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250627 13:54:33.575232 14624 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.20: config changed from index 19 to 20, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:33.575778 8189 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index 19 to 20, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:33.577898 14730 ts_tablet_manager.cc:927] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096: Initiating tablet copy from peer 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101)
I20250627 13:54:33.582500 7652 catalog_manager.cc:5582] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 19 to 20, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:33.587510 7927 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index 19 to 20, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:33.598099 8063 consensus_queue.cc:237] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 1.20, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } }
I20250627 13:54:33.600467 7801 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index 19 to 20, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:33.602609 8189 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 1 index: 21. (index mismatch)
I20250627 13:54:33.604327 14597 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250627 13:54:33.604602 14730 tablet_copy_client.cc:323] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.69:33101
I20250627 13:54:33.605321 9731 tablet_copy_service.cc:140] P 2a1282ebe92a4c9a92eed3b13f69012a: Received BeginTabletCopySession request for tablet e3053b529a264e8697a1df9e17eeea81 from peer 83574c1bf8ea45adaff939158f93c096 ({username='slave'} at 127.0.103.65:56847)
I20250627 13:54:33.605522 9731 tablet_copy_service.cc:161] P 2a1282ebe92a4c9a92eed3b13f69012a: Beginning new tablet copy session on tablet e3053b529a264e8697a1df9e17eeea81 from peer 83574c1bf8ea45adaff939158f93c096 at {username='slave'} at 127.0.103.65:56847: session id = 83574c1bf8ea45adaff939158f93c096-e3053b529a264e8697a1df9e17eeea81
I20250627 13:54:33.609318 9731 tablet_copy_source_session.cc:215] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:33.610704 14730 ts_tablet_manager.cc:1905] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250627 13:54:33.614307 14585 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.21: config changed from index 20 to 21, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
W20250627 13:54:33.618410 7866 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.70:40647, user_credentials={real_user=slave}} blocked reactor thread for 159821us
I20250627 13:54:33.622184 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 644985e3086d4614b6598a727cb60c1d with cas_config_opid_index 20: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:33.624276 7800 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 1 index: 21. (index mismatch)
I20250627 13:54:33.628279 7652 catalog_manager.cc:5582] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 20 to 21, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:33.631335 14585 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250627 13:54:33.632977 7800 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.21: config changed from index 20 to 21, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:33.633785 14730 ts_tablet_manager.cc:1918] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20250627 13:54:33.634080 14730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3053b529a264e8697a1df9e17eeea81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:33.635423 7911 tablet_service.cc:1515] Processing DeleteTablet for tablet 644985e3086d4614b6598a727cb60c1d with delete_type TABLET_DATA_TOMBSTONED (TS a41a89ae8ed743edb2b5cf4490d8c854 not found in new config with opid_index 21) from {username='slave'} at 127.0.0.1:59302
I20250627 13:54:33.644572 8189 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.21: config changed from index 20 to 21, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:33.653425 14730 tablet_copy_client.cc:806] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:33.653743 14730 tablet_copy_client.cc:670] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:33.655750 14730 tablet_copy_client.cc:538] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:33.666558 14753 tablet_replica.cc:331] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854: stopping tablet replica
I20250627 13:54:33.666858 14753 raft_consensus.cc:2241] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:33.667214 14753 raft_consensus.cc:2270] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:33.667946 14730 tablet_bootstrap.cc:492] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:54:33.674453 14753 ts_tablet_manager.cc:1905] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:33.697805 14753 ts_tablet_manager.cc:1918] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.20
I20250627 13:54:33.698246 14753 log.cc:1199] T 644985e3086d4614b6598a727cb60c1d P a41a89ae8ed743edb2b5cf4490d8c854: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/wals/644985e3086d4614b6598a727cb60c1d
I20250627 13:54:33.699404 7647 catalog_manager.cc:4928] TS a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675): tablet 644985e3086d4614b6598a727cb60c1d (table rebalance_test_table_1 [id=8961aa0df2f1488f82cd55cfa22ad214]) successfully deleted
I20250627 13:54:33.751739 14762 ts_tablet_manager.cc:927] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:33.762264 14761 ts_tablet_manager.cc:927] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:33.763353 14762 tablet_copy_client.cc:323] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:33.764271 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet e2bd060a4a9e4765b45b0242ae56e489 from peer 83574c1bf8ea45adaff939158f93c096 ({username='slave'} at 127.0.103.65:50643)
I20250627 13:54:33.764477 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet e2bd060a4a9e4765b45b0242ae56e489 from peer 83574c1bf8ea45adaff939158f93c096 at {username='slave'} at 127.0.103.65:50643: session id = 83574c1bf8ea45adaff939158f93c096-e2bd060a4a9e4765b45b0242ae56e489
I20250627 13:54:33.769496 8082 tablet_copy_source_session.cc:215] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:33.779814 14762 ts_tablet_manager.cc:1905] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250627 13:54:33.780148 14761 tablet_copy_client.cc:323] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:33.781796 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet e2bd060a4a9e4765b45b0242ae56e489 from peer 956ff058e6d64579a1963aabdb6d466b ({username='slave'} at 127.0.103.70:59981)
I20250627 13:54:33.782049 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet e2bd060a4a9e4765b45b0242ae56e489 from peer 956ff058e6d64579a1963aabdb6d466b at {username='slave'} at 127.0.103.70:59981: session id = 956ff058e6d64579a1963aabdb6d466b-e2bd060a4a9e4765b45b0242ae56e489
I20250627 13:54:33.785941 8082 tablet_copy_source_session.cc:215] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:33.795467 14761 ts_tablet_manager.cc:1905] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250627 13:54:33.802273 14761 ts_tablet_manager.cc:1918] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.11
I20250627 13:54:33.802521 14761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2bd060a4a9e4765b45b0242ae56e489. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:33.806263 14761 tablet_copy_client.cc:806] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:33.806478 14761 tablet_copy_client.cc:670] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:33.809633 14761 tablet_copy_client.cc:538] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:33.813272 14761 tablet_bootstrap.cc:492] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap starting.
I20250627 13:54:33.840451 14762 ts_tablet_manager.cc:1918] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20250627 13:54:33.848496 14762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2bd060a4a9e4765b45b0242ae56e489. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:33.869717 14762 tablet_copy_client.cc:806] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:33.870095 14762 tablet_copy_client.cc:670] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:33.882572 14720 tablet_bootstrap.cc:492] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:33.882964 14720 tablet_bootstrap.cc:492] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: Bootstrap complete.
I20250627 13:54:33.883219 14720 ts_tablet_manager.cc:1397] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.355s user 0.044s sys 0.029s
I20250627 13:54:33.883996 14720 raft_consensus.cc:357] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } }
I20250627 13:54:33.884210 14720 raft_consensus.cc:738] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: LEARNER
I20250627 13:54:33.884383 14720 consensus_queue.cc:260] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } }
I20250627 13:54:33.885701 14720 ts_tablet_manager.cc:1428] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250627 13:54:33.886325 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session 83574c1bf8ea45adaff939158f93c096-d2eb537d49804e1b8ec5a217b1564988 received from {username='slave'} at 127.0.103.65:50643
I20250627 13:54:33.886489 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session 83574c1bf8ea45adaff939158f93c096-d2eb537d49804e1b8ec5a217b1564988 on tablet d2eb537d49804e1b8ec5a217b1564988 with peer 83574c1bf8ea45adaff939158f93c096
I20250627 13:54:33.893079 14762 tablet_copy_client.cc:538] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:33.896569 14762 tablet_bootstrap.cc:492] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096: Bootstrap starting.
I20250627 13:54:33.913219 14730 tablet_bootstrap.cc:492] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096: Bootstrap replayed 1/1 log segments. Stats: ops{read=22 overwritten=0 applied=22 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:33.930152 14730 tablet_bootstrap.cc:492] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096: Bootstrap complete.
I20250627 13:54:33.930501 14730 ts_tablet_manager.cc:1397] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.264s user 0.090s sys 0.012s
I20250627 13:54:33.931363 14730 raft_consensus.cc:357] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } }
I20250627 13:54:33.931633 14730 raft_consensus.cc:738] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: LEARNER
I20250627 13:54:33.931851 14730 consensus_queue.cc:260] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 22, Last appended: 2.22, Last appended by leader: 22, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } }
I20250627 13:54:33.956517 14730 ts_tablet_manager.cc:1428] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20250627 13:54:33.957147 9731 tablet_copy_service.cc:342] P 2a1282ebe92a4c9a92eed3b13f69012a: Request end of tablet copy session 83574c1bf8ea45adaff939158f93c096-e3053b529a264e8697a1df9e17eeea81 received from {username='slave'} at 127.0.103.65:56847
I20250627 13:54:33.957281 9731 tablet_copy_service.cc:434] P 2a1282ebe92a4c9a92eed3b13f69012a: ending tablet copy session 83574c1bf8ea45adaff939158f93c096-e3053b529a264e8697a1df9e17eeea81 on tablet e3053b529a264e8697a1df9e17eeea81 with peer 83574c1bf8ea45adaff939158f93c096
I20250627 13:54:33.963064 14761 tablet_bootstrap.cc:492] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:33.963562 14761 tablet_bootstrap.cc:492] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b: Bootstrap complete.
I20250627 13:54:33.963907 14761 ts_tablet_manager.cc:1397] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b: Time spent bootstrapping tablet: real 0.151s user 0.060s sys 0.003s
I20250627 13:54:33.964944 14761 raft_consensus.cc:357] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:33.965202 14761 raft_consensus.cc:738] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 956ff058e6d64579a1963aabdb6d466b, State: Initialized, Role: LEARNER
I20250627 13:54:33.965430 14761 consensus_queue.cc:260] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 1.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:33.966018 14761 ts_tablet_manager.cc:1428] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20250627 13:54:33.966934 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session 956ff058e6d64579a1963aabdb6d466b-e2bd060a4a9e4765b45b0242ae56e489 received from {username='slave'} at 127.0.103.70:59981
I20250627 13:54:33.967063 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session 956ff058e6d64579a1963aabdb6d466b-e2bd060a4a9e4765b45b0242ae56e489 on tablet e2bd060a4a9e4765b45b0242ae56e489 with peer 956ff058e6d64579a1963aabdb6d466b
I20250627 13:54:34.143934 9840 raft_consensus.cc:1215] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.13->[1.14-1.14] Dedup: 1.14->[]
I20250627 13:54:34.194849 14762 tablet_bootstrap.cc:492] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:34.195412 14762 tablet_bootstrap.cc:492] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096: Bootstrap complete.
I20250627 13:54:34.195819 14762 ts_tablet_manager.cc:1397] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096: Time spent bootstrapping tablet: real 0.299s user 0.061s sys 0.008s
I20250627 13:54:34.197631 14762 raft_consensus.cc:357] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:34.198068 14762 raft_consensus.cc:738] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83574c1bf8ea45adaff939158f93c096, State: Initialized, Role: LEARNER
I20250627 13:54:34.198359 14762 consensus_queue.cc:260] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 1.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:34.200711 14762 ts_tablet_manager.cc:1428] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250627 13:54:34.202351 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session 83574c1bf8ea45adaff939158f93c096-e2bd060a4a9e4765b45b0242ae56e489 received from {username='slave'} at 127.0.103.65:50643
I20250627 13:54:34.202499 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session 83574c1bf8ea45adaff939158f93c096-e2bd060a4a9e4765b45b0242ae56e489 on tablet e2bd060a4a9e4765b45b0242ae56e489 with peer 83574c1bf8ea45adaff939158f93c096
I20250627 13:54:34.211118 7800 raft_consensus.cc:1215] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.13->[1.14-1.14] Dedup: 1.14->[]
I20250627 13:54:34.250767 7800 raft_consensus.cc:1215] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.15->[1.16-1.16] Dedup: 1.16->[]
I20250627 13:54:34.320159 9691 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:34.345692 9691 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:34.423410 7800 raft_consensus.cc:1215] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.21->[2.22-2.22] Dedup: 2.22->[]
I20250627 13:54:34.425457 9691 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:34.464464 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:34.470005 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:34.515751 9822 tablet_service.cc:1430] Tablet server has 3 leaders and 0 scanners
I20250627 13:54:34.526530 7782 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:34.540011 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:34.552210 7911 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:34.576524 8041 tablet_service.cc:1430] Tablet server has 8 leaders and 0 scanners
I20250627 13:54:34.599473 7782 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:34.602849 8041 tablet_service.cc:1430] Tablet server has 8 leaders and 0 scanners
I20250627 13:54:34.627012 7782 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:34.633572 9822 tablet_service.cc:1430] Tablet server has 3 leaders and 0 scanners
I20250627 13:54:34.656046 10083 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:34.659148 8041 tablet_service.cc:1430] Tablet server has 8 leaders and 0 scanners
I20250627 13:54:34.660799 14624 raft_consensus.cc:1062] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER 956ff058e6d64579a1963aabdb6d466b to VOTER
I20250627 13:54:34.662086 14624 consensus_queue.cc:237] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:34.668815 9822 tablet_service.cc:1430] Tablet server has 3 leaders and 0 scanners
I20250627 13:54:34.703964 7911 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:34.723176 9840 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250627 13:54:34.723925 14597 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250627 13:54:34.733727 7800 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250627 13:54:34.734248 14597 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250627 13:54:34.741257 10083 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:34.747330 8171 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:34.759119 8189 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250627 13:54:34.759797 14597 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250627 13:54:34.768807 14754 raft_consensus.cc:1062] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER 83574c1bf8ea45adaff939158f93c096 to VOTER
I20250627 13:54:34.769608 14754 consensus_queue.cc:237] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } }
I20250627 13:54:34.786448 9709 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250627 13:54:34.787060 14597 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250627 13:54:34.799026 7801 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250627 13:54:34.799736 14597 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250627 13:54:34.800460 8190 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250627 13:54:34.807121 10102 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250627 13:54:34.807793 14754 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250627 13:54:34.808132 14597 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250627 13:54:34.809772 7911 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:34.853811 14624 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.17: config changed from index 16 to 17, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:34.858099 14754 raft_consensus.cc:1025] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: attempt to promote peer 83574c1bf8ea45adaff939158f93c096: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250627 13:54:34.861680 8171 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:34.866958 14754 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:34.872443 9840 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:34.883716 7652 catalog_manager.cc:5582] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b reported cstate change: config changed from index 14 to 15, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:34.886778 10083 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:34.887501 8190 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:34.888269 7800 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:34.890555 8192 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:34.905555 7659 catalog_manager.cc:5582] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 16 to 17, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:34.908077 8171 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:34.930899 14796 raft_consensus.cc:1062] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a: attempting to promote NON_VOTER 83574c1bf8ea45adaff939158f93c096 to VOTER
I20250627 13:54:34.931761 14796 consensus_queue.cc:237] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 2.22, Last appended by leader: 16, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } }
I20250627 13:54:34.939740 9709 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:34.939900 7801 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEARNER]: Committing config change with OpId 1.15: config changed from index 14 to 15, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: NON_VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:34.946236 14597 raft_consensus.cc:1062] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER 83574c1bf8ea45adaff939158f93c096 to VOTER
I20250627 13:54:34.947196 14597 consensus_queue.cc:237] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:34.950997 9840 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20250627 13:54:34.951251 7802 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20250627 13:54:34.956832 14754 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250627 13:54:34.958040 14754 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250627 13:54:34.963798 14754 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:34.969913 7801 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:34.998869 10102 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:35.009835 7658 catalog_manager.cc:5582] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 15 to 16, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:35.013263 8063 consensus_queue.cc:237] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 1.17, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } }
I20250627 13:54:35.015326 7801 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20250627 13:54:35.015455 10102 raft_consensus.cc:1273] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20250627 13:54:35.015964 14942 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250627 13:54:35.017474 14957 consensus_queue.cc:1035] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250627 13:54:35.020463 14624 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.18: config changed from index 17 to 18, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:35.022372 10102 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:35.024596 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet d2eb537d49804e1b8ec5a217b1564988 with cas_config_opid_index 17: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:35.026355 7801 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 [term 2 LEARNER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 22. Preceding OpId from leader: term: 2 index: 23. (index mismatch)
I20250627 13:54:35.027530 14568 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20250627 13:54:35.028738 7658 catalog_manager.cc:5582] T d2eb537d49804e1b8ec5a217b1564988 P dede81a8c0164427a61532d01b6fc157 reported cstate change: config changed from index 17 to 18, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:35.032676 8171 tablet_service.cc:1515] Processing DeleteTablet for tablet d2eb537d49804e1b8ec5a217b1564988 with delete_type TABLET_DATA_TOMBSTONED (TS e4c831df4da841929d13d6c783099b0e not found in new config with opid_index 18) from {username='slave'} at 127.0.0.1:48616
I20250627 13:54:35.033280 14963 tablet_replica.cc:331] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e: stopping tablet replica
I20250627 13:54:35.033543 14963 raft_consensus.cc:2241] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:35.033844 14963 raft_consensus.cc:2270] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:35.034796 14963 ts_tablet_manager.cc:1905] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:35.041009 14963 ts_tablet_manager.cc:1918] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.17
I20250627 13:54:35.041188 14963 log.cc:1199] T d2eb537d49804e1b8ec5a217b1564988 P e4c831df4da841929d13d6c783099b0e: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal/wals/d2eb537d49804e1b8ec5a217b1564988
I20250627 13:54:35.041971 7648 catalog_manager.cc:4928] TS e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): tablet d2eb537d49804e1b8ec5a217b1564988 (table rebalance_test_table_0 [id=d8bfc9268a4a4c2c92bc20d56ac73086]) successfully deleted
I20250627 13:54:35.048741 7800 raft_consensus.cc:2953] T d2eb537d49804e1b8ec5a217b1564988 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:35.050690 8189 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20250627 13:54:35.051311 14957 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250627 13:54:35.051416 9840 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:35.053234 8190 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:35.053864 9709 raft_consensus.cc:2953] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:35.071079 8063 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "e2bd060a4a9e4765b45b0242ae56e489"
dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35304
I20250627 13:54:35.071264 8063 raft_consensus.cc:604] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Received request to transfer leadership
W20250627 13:54:35.095945 7645 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leader transfer in progress
I20250627 13:54:35.115278 8189 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 22. Preceding OpId from leader: term: 2 index: 23. (index mismatch)
I20250627 13:54:35.116369 9973 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 22. Preceding OpId from leader: term: 2 index: 23. (index mismatch)
I20250627 13:54:35.117297 14960 raft_consensus.cc:991] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660: : Instructing follower 2a1282ebe92a4c9a92eed3b13f69012a to start an election
I20250627 13:54:35.117461 14960 raft_consensus.cc:1079] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Signalling peer 2a1282ebe92a4c9a92eed3b13f69012a to start an election
I20250627 13:54:35.117548 14796 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20250627 13:54:35.119427 14796 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
W20250627 13:54:35.124742 7645 catalog_manager.cc:4671] TS 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933): ChangeConfig:REMOVE_PEER RPC failed for tablet e2bd060a4a9e4765b45b0242ae56e489: Remote error: Service unavailable: leader transfer in progress
I20250627 13:54:35.127463 9709 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "e2bd060a4a9e4765b45b0242ae56e489"
dest_uuid: "2a1282ebe92a4c9a92eed3b13f69012a"
from {username='slave'} at 127.0.103.67:60633
I20250627 13:54:35.127696 9709 raft_consensus.cc:491] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:54:35.128355 9709 raft_consensus.cc:3058] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:35.134766 9709 raft_consensus.cc:513] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:35.136788 14961 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 LEADER]: Committing config change with OpId 2.23: config changed from index 22 to 23, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:35.141443 7801 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Committing config change with OpId 2.23: config changed from index 22 to 23, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:35.147567 7658 catalog_manager.cc:5582] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 22 to 23, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:35.148819 9973 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Committing config change with OpId 2.23: config changed from index 22 to 23, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:35.159286 8189 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Committing config change with OpId 2.23: config changed from index 22 to 23, 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) changed from NON_VOTER to VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:35.171551 9708 consensus_queue.cc:237] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 2.23, Last appended by leader: 16, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } }
I20250627 13:54:35.173496 7801 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 23. Preceding OpId from leader: term: 2 index: 24. (index mismatch)
I20250627 13:54:35.174150 14796 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20250627 13:54:35.186614 8063 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2bd060a4a9e4765b45b0242ae56e489" candidate_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: true dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660"
I20250627 13:54:35.186815 8063 raft_consensus.cc:3053] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Stepping down as leader of term 1
I20250627 13:54:35.186892 8063 raft_consensus.cc:738] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9cf479bdcd0242738d2a77db4ad9f660, State: Running, Role: LEADER
I20250627 13:54:35.187170 8063 consensus_queue.cc:260] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:35.187563 8063 raft_consensus.cc:3058] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:35.193420 8063 raft_consensus.cc:2466] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1282ebe92a4c9a92eed3b13f69012a in term 2.
I20250627 13:54:35.194653 7800 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2bd060a4a9e4765b45b0242ae56e489" candidate_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: true dest_uuid: "83574c1bf8ea45adaff939158f93c096"
I20250627 13:54:35.194813 7800 raft_consensus.cc:3058] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:35.197381 7800 raft_consensus.cc:2466] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1282ebe92a4c9a92eed3b13f69012a in term 2.
I20250627 13:54:35.197980 9644 leader_election.cc:304] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2a1282ebe92a4c9a92eed3b13f69012a, 83574c1bf8ea45adaff939158f93c096, 9cf479bdcd0242738d2a77db4ad9f660; no voters:
I20250627 13:54:35.198395 14961 raft_consensus.cc:2802] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:54:35.198568 14961 raft_consensus.cc:695] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 LEADER]: Becoming Leader. State: Replica: 2a1282ebe92a4c9a92eed3b13f69012a, State: Running, Role: LEADER
I20250627 13:54:35.198908 14961 consensus_queue.cc:237] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:35.201787 14796 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 LEADER]: Committing config change with OpId 2.24: config changed from index 23 to 24, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:35.205940 7801 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Committing config change with OpId 2.24: config changed from index 23 to 24, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:35.211395 7659 catalog_manager.cc:5582] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 23 to 24, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New cstate: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:35.211475 7658 catalog_manager.cc:5582] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: term changed from 1 to 2, leader changed from 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67) to 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69). New cstate: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:35.214087 8190 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2bd060a4a9e4765b45b0242ae56e489" candidate_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: true dest_uuid: "e4c831df4da841929d13d6c783099b0e"
I20250627 13:54:35.214249 8190 raft_consensus.cc:3058] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:35.216939 8190 raft_consensus.cc:2466] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1282ebe92a4c9a92eed3b13f69012a in term 2.
I20250627 13:54:35.217729 8171 tablet_service.cc:1515] Processing DeleteTablet for tablet e3053b529a264e8697a1df9e17eeea81 with delete_type TABLET_DATA_TOMBSTONED (TS e4c831df4da841929d13d6c783099b0e not found in new config with opid_index 24) from {username='slave'} at 127.0.0.1:48616
W20250627 13:54:35.221239 7645 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet e2bd060a4a9e4765b45b0242ae56e489 with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER failed with leader 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 9cf479bdcd0242738d2a77db4ad9f660 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:35.221706 7648 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet e3053b529a264e8697a1df9e17eeea81 with cas_config_opid_index 23: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:35.225035 14963 tablet_replica.cc:331] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: stopping tablet replica
I20250627 13:54:35.225287 14963 raft_consensus.cc:2241] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:35.225507 14963 raft_consensus.cc:2270] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:35.228428 14963 ts_tablet_manager.cc:1905] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:35.238157 14963 ts_tablet_manager.cc:1918] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.23
I20250627 13:54:35.239436 14963 log.cc:1199] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal/wals/e3053b529a264e8697a1df9e17eeea81
I20250627 13:54:35.240329 7648 catalog_manager.cc:4928] TS e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): tablet e3053b529a264e8697a1df9e17eeea81 (table rebalance_test_table_0 [id=d8bfc9268a4a4c2c92bc20d56ac73086]) successfully deleted
I20250627 13:54:35.244935 8171 tablet_service.cc:1515] Processing DeleteTablet for tablet e3053b529a264e8697a1df9e17eeea81 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 23 (current committed config index is 24)) from {username='slave'} at 127.0.0.1:48616
I20250627 13:54:35.245405 7648 catalog_manager.cc:4928] TS e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): tablet e3053b529a264e8697a1df9e17eeea81 (table rebalance_test_table_0 [id=d8bfc9268a4a4c2c92bc20d56ac73086]) successfully deleted
I20250627 13:54:35.247480 9709 leader_election.cc:290] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [CANDIDATE]: Term 2 election: Requested vote from peers 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933), e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467), 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647)
I20250627 13:54:35.251061 9974 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 23. Preceding OpId from leader: term: 2 index: 24. (index mismatch)
I20250627 13:54:35.251879 14961 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20250627 13:54:35.253820 9974 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Committing config change with OpId 2.24: config changed from index 23 to 24, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } }
I20250627 13:54:35.283380 9840 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e2bd060a4a9e4765b45b0242ae56e489" candidate_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: true dest_uuid: "956ff058e6d64579a1963aabdb6d466b"
I20250627 13:54:35.283638 9840 raft_consensus.cc:3058] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:35.286628 9840 raft_consensus.cc:2466] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a1282ebe92a4c9a92eed3b13f69012a in term 2.
I20250627 13:54:35.413478 8189 consensus_queue.cc:237] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:35.416183 10102 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250627 13:54:35.416246 8063 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
W20250627 13:54:35.416536 8126 consensus_peers.cc:487] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:35.416800 14716 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250627 13:54:35.417248 14716 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250627 13:54:35.421702 8062 consensus_queue.cc:237] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 1.21, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
W20250627 13:54:35.436022 7996 consensus_peers.cc:487] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:35.437119 7801 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 1 index: 22. (index mismatch)
I20250627 13:54:35.437656 8189 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 1 index: 22. (index mismatch)
I20250627 13:54:35.437922 14954 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250627 13:54:35.438221 14624 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250627 13:54:35.439605 14977 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:35.447074 7659 catalog_manager.cc:5582] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 14 to 15, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:35.453735 10102 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:35.464212 14624 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.22: config changed from index 21 to 22, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:35.464949 7800 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.22: config changed from index 21 to 22, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:35.471676 7659 catalog_manager.cc:5582] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 21 to 22, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:35.496832 8063 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:35.521154 8189 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.22: config changed from index 21 to 22, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:35.595304 7801 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 2 index: 17. (index mismatch)
I20250627 13:54:35.595775 14796 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250627 13:54:35.654646 8189 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 2 index: 17. (index mismatch)
I20250627 13:54:35.655710 14796 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250627 13:54:35.667470 9840 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 2 index: 17. (index mismatch)
I20250627 13:54:35.668058 8063 raft_consensus.cc:1273] T e2bd060a4a9e4765b45b0242ae56e489 P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 2 index: 17. (index mismatch)
I20250627 13:54:35.668975 14568 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250627 13:54:35.669420 14796 consensus_queue.cc:1035] T e2bd060a4a9e4765b45b0242ae56e489 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250627 13:54:35.867465 15062 ts_tablet_manager.cc:927] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:35.868207 15062 tablet_copy_client.cc:323] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:35.869006 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet 644985e3086d4614b6598a727cb60c1d from peer 956ff058e6d64579a1963aabdb6d466b ({username='slave'} at 127.0.103.70:59981)
I20250627 13:54:35.869144 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet 644985e3086d4614b6598a727cb60c1d from peer 956ff058e6d64579a1963aabdb6d466b at {username='slave'} at 127.0.103.70:59981: session id = 956ff058e6d64579a1963aabdb6d466b-644985e3086d4614b6598a727cb60c1d
I20250627 13:54:35.871541 8082 tablet_copy_source_session.cc:215] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:35.872617 15062 ts_tablet_manager.cc:1905] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250627 13:54:35.878836 15062 ts_tablet_manager.cc:1918] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.13
I20250627 13:54:35.879110 15062 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 644985e3086d4614b6598a727cb60c1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:35.882187 15062 tablet_copy_client.cc:806] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:35.882396 15062 tablet_copy_client.cc:670] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:35.884217 15062 tablet_copy_client.cc:538] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:35.886688 15062 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: Bootstrap starting.
I20250627 13:54:35.934897 15062 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: Bootstrap replayed 1/1 log segments. Stats: ops{read=22 overwritten=0 applied=22 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:35.935263 15062 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: Bootstrap complete.
I20250627 13:54:35.935487 15062 ts_tablet_manager.cc:1397] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: Time spent bootstrapping tablet: real 0.049s user 0.047s sys 0.000s
I20250627 13:54:35.936123 15062 raft_consensus.cc:357] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:35.936323 15062 raft_consensus.cc:738] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 956ff058e6d64579a1963aabdb6d466b, State: Initialized, Role: LEARNER
I20250627 13:54:35.936532 15062 consensus_queue.cc:260] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 22, Last appended: 1.22, Last appended by leader: 22, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:35.937101 15062 ts_tablet_manager.cc:1428] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250627 13:54:35.937693 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session 956ff058e6d64579a1963aabdb6d466b-644985e3086d4614b6598a727cb60c1d received from {username='slave'} at 127.0.103.70:59981
I20250627 13:54:35.937832 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session 956ff058e6d64579a1963aabdb6d466b-644985e3086d4614b6598a727cb60c1d on tablet 644985e3086d4614b6598a727cb60c1d with peer 956ff058e6d64579a1963aabdb6d466b
I20250627 13:54:35.947228 15062 ts_tablet_manager.cc:927] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: Initiating tablet copy from peer e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:54:35.947829 15062 tablet_copy_client.cc:323] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.68:46139
I20250627 13:54:35.948488 8213 tablet_copy_service.cc:140] P e4c831df4da841929d13d6c783099b0e: Received BeginTabletCopySession request for tablet 14dba72d45694defb83f41c1572722ca from peer 956ff058e6d64579a1963aabdb6d466b ({username='slave'} at 127.0.103.70:33381)
I20250627 13:54:35.948623 8213 tablet_copy_service.cc:161] P e4c831df4da841929d13d6c783099b0e: Beginning new tablet copy session on tablet 14dba72d45694defb83f41c1572722ca from peer 956ff058e6d64579a1963aabdb6d466b at {username='slave'} at 127.0.103.70:33381: session id = 956ff058e6d64579a1963aabdb6d466b-14dba72d45694defb83f41c1572722ca
I20250627 13:54:35.950932 8213 tablet_copy_source_session.cc:215] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:35.951717 15062 ts_tablet_manager.cc:1905] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250627 13:54:35.956290 15062 ts_tablet_manager.cc:1918] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.13
I20250627 13:54:35.956540 15062 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14dba72d45694defb83f41c1572722ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:35.959609 15062 tablet_copy_client.cc:806] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:35.959805 15062 tablet_copy_client.cc:670] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:35.961668 15062 tablet_copy_client.cc:538] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:35.964102 15062 tablet_bootstrap.cc:492] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: Bootstrap starting.
I20250627 13:54:36.006654 15062 tablet_bootstrap.cc:492] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:36.007026 15062 tablet_bootstrap.cc:492] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: Bootstrap complete.
I20250627 13:54:36.007248 15062 ts_tablet_manager.cc:1397] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: Time spent bootstrapping tablet: real 0.043s user 0.032s sys 0.010s
I20250627 13:54:36.007903 15062 raft_consensus.cc:357] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:36.008074 15062 raft_consensus.cc:738] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 956ff058e6d64579a1963aabdb6d466b, State: Initialized, Role: LEARNER
I20250627 13:54:36.008302 15062 consensus_queue.cc:260] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 1.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:36.008858 15062 ts_tablet_manager.cc:1428] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250627 13:54:36.009660 8213 tablet_copy_service.cc:342] P e4c831df4da841929d13d6c783099b0e: Request end of tablet copy session 956ff058e6d64579a1963aabdb6d466b-14dba72d45694defb83f41c1572722ca received from {username='slave'} at 127.0.103.70:33381
I20250627 13:54:36.009776 8213 tablet_copy_service.cc:434] P e4c831df4da841929d13d6c783099b0e: ending tablet copy session 956ff058e6d64579a1963aabdb6d466b-14dba72d45694defb83f41c1572722ca on tablet 14dba72d45694defb83f41c1572722ca with peer 956ff058e6d64579a1963aabdb6d466b
W20250627 13:54:36.344347 7648 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet e2bd060a4a9e4765b45b0242ae56e489 with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER failed with leader 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101) due to error UNKNOWN_ERROR; will retry: Invalid argument: Cannot remove peer 2a1282ebe92a4c9a92eed3b13f69012a from the config because it is the leader. Force another leader to be elected to remove this peer. Consensus state: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:36.387053 9840 raft_consensus.cc:1215] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.21->[1.22-1.22] Dedup: 1.22->[]
I20250627 13:54:36.392349 9840 raft_consensus.cc:1215] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.14->[1.15-1.15] Dedup: 1.15->[]
I20250627 13:54:36.919322 15172 raft_consensus.cc:1062] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e: attempting to promote NON_VOTER 956ff058e6d64579a1963aabdb6d466b to VOTER
I20250627 13:54:36.920102 15172 consensus_queue.cc:237] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:36.922049 9840 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20250627 13:54:36.922158 8063 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20250627 13:54:36.922675 15174 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250627 13:54:36.922758 10103 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20250627 13:54:36.922905 15119 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250627 13:54:36.923246 15172 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250627 13:54:36.926154 15119 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:36.926688 9840 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:36.931135 7658 catalog_manager.cc:5582] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b reported cstate change: config changed from index 15 to 16, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:36.933571 8063 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:36.948082 10102 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:36.957983 15181 raft_consensus.cc:1062] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER 956ff058e6d64579a1963aabdb6d466b to VOTER
I20250627 13:54:36.959143 15181 consensus_queue.cc:237] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 1.22, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:36.963801 7801 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 1 index: 23. (index mismatch)
I20250627 13:54:36.963809 8189 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 1 index: 23. (index mismatch)
I20250627 13:54:36.964390 9840 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 1 index: 23. (index mismatch)
I20250627 13:54:36.964382 14624 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20250627 13:54:36.964802 15180 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20250627 13:54:36.965094 15181 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20250627 13:54:36.968353 15181 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.23: config changed from index 22 to 23, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:36.969034 9840 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.23: config changed from index 22 to 23, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:36.969470 7801 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.23: config changed from index 22 to 23, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:36.973116 8189 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.23: config changed from index 22 to 23, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:36.974129 7652 catalog_manager.cc:5582] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 22 to 23, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false replace: true } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:36.980247 8063 consensus_queue.cc:237] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:36.982038 7801 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20250627 13:54:36.982230 9840 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20250627 13:54:36.982537 15180 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20250627 13:54:36.982806 15181 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20250627 13:54:36.984798 14624 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.24: config changed from index 23 to 24, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:36.985735 9840 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index 23 to 24, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:36.985821 7801 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index 23 to 24, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:36.987620 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 644985e3086d4614b6598a727cb60c1d with cas_config_opid_index 23: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:36.989383 7661 catalog_manager.cc:5582] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 23 to 24, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:36.992954 8171 tablet_service.cc:1515] Processing DeleteTablet for tablet 644985e3086d4614b6598a727cb60c1d with delete_type TABLET_DATA_TOMBSTONED (TS e4c831df4da841929d13d6c783099b0e not found in new config with opid_index 24) from {username='slave'} at 127.0.0.1:48616
I20250627 13:54:36.994086 15191 tablet_replica.cc:331] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: stopping tablet replica
I20250627 13:54:36.994330 15191 raft_consensus.cc:2241] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:36.994596 15191 raft_consensus.cc:2270] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:36.995383 15191 ts_tablet_manager.cc:1905] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:37.000835 15191 ts_tablet_manager.cc:1918] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.23
I20250627 13:54:37.000931 15191 log.cc:1199] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal/wals/644985e3086d4614b6598a727cb60c1d
I20250627 13:54:37.001608 7648 catalog_manager.cc:4928] TS e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): tablet 644985e3086d4614b6598a727cb60c1d (table rebalance_test_table_1 [id=8961aa0df2f1488f82cd55cfa22ad214]) successfully deleted
I20250627 13:54:37.052268 8189 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "14dba72d45694defb83f41c1572722ca"
dest_uuid: "e4c831df4da841929d13d6c783099b0e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37324
I20250627 13:54:37.052464 8189 raft_consensus.cc:604] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Received request to transfer leadership
I20250627 13:54:37.149451 9691 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:37.159482 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:37.183241 8041 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:37.183322 7911 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:37.190048 8171 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:37.202399 10083 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:37.216517 7782 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:37.224659 9822 tablet_service.cc:1430] Tablet server has 3 leaders and 0 scanners
I20250627 13:54:37.367985 15178 raft_consensus.cc:991] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e: : Instructing follower 956ff058e6d64579a1963aabdb6d466b to start an election
I20250627 13:54:37.368134 15178 raft_consensus.cc:1079] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Signalling peer 956ff058e6d64579a1963aabdb6d466b to start an election
I20250627 13:54:37.368842 9840 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "14dba72d45694defb83f41c1572722ca"
dest_uuid: "956ff058e6d64579a1963aabdb6d466b"
from {username='slave'} at 127.0.103.68:50533
I20250627 13:54:37.369053 9840 raft_consensus.cc:491] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:54:37.369140 9840 raft_consensus.cc:3058] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:37.372018 9840 raft_consensus.cc:513] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:37.373548 8189 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14dba72d45694defb83f41c1572722ca" candidate_uuid: "956ff058e6d64579a1963aabdb6d466b" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: true dest_uuid: "e4c831df4da841929d13d6c783099b0e"
I20250627 13:54:37.373733 8189 raft_consensus.cc:3053] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Stepping down as leader of term 1
I20250627 13:54:37.373790 8189 raft_consensus.cc:738] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Running, Role: LEADER
I20250627 13:54:37.374202 8189 consensus_queue.cc:260] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:37.374605 8189 raft_consensus.cc:3058] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:37.377413 8189 raft_consensus.cc:2466] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 956ff058e6d64579a1963aabdb6d466b in term 2.
I20250627 13:54:37.380431 9840 leader_election.cc:290] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [CANDIDATE]: Term 2 election: Requested vote from peers e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139), dede81a8c0164427a61532d01b6fc157 (127.0.103.72:35213), 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:37.382149 8063 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14dba72d45694defb83f41c1572722ca" candidate_uuid: "956ff058e6d64579a1963aabdb6d466b" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: true dest_uuid: "9cf479bdcd0242738d2a77db4ad9f660"
I20250627 13:54:37.382447 8063 raft_consensus.cc:3058] T 14dba72d45694defb83f41c1572722ca P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:37.387411 8063 raft_consensus.cc:2466] T 14dba72d45694defb83f41c1572722ca P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 956ff058e6d64579a1963aabdb6d466b in term 2.
I20250627 13:54:37.387955 9775 leader_election.cc:304] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 956ff058e6d64579a1963aabdb6d466b, 9cf479bdcd0242738d2a77db4ad9f660, e4c831df4da841929d13d6c783099b0e; no voters:
I20250627 13:54:37.388307 14808 raft_consensus.cc:2802] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:54:37.388417 14808 raft_consensus.cc:695] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [term 2 LEADER]: Becoming Leader. State: Replica: 956ff058e6d64579a1963aabdb6d466b, State: Running, Role: LEADER
I20250627 13:54:37.388765 14808 consensus_queue.cc:237] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:37.396780 7661 catalog_manager.cc:5582] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b reported cstate change: term changed from 1 to 2, leader changed from e4c831df4da841929d13d6c783099b0e (127.0.103.68) to 956ff058e6d64579a1963aabdb6d466b (127.0.103.70). New cstate: current_term: 2 leader_uuid: "956ff058e6d64579a1963aabdb6d466b" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250627 13:54:37.419358 7648 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet e2bd060a4a9e4765b45b0242ae56e489 with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER failed with leader 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101) due to error UNKNOWN_ERROR; will retry: Invalid argument: Cannot remove peer 2a1282ebe92a4c9a92eed3b13f69012a from the config because it is the leader. Force another leader to be elected to remove this peer. Consensus state: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:37.421978 10103 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14dba72d45694defb83f41c1572722ca" candidate_uuid: "956ff058e6d64579a1963aabdb6d466b" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: true dest_uuid: "dede81a8c0164427a61532d01b6fc157"
I20250627 13:54:37.422217 10103 raft_consensus.cc:3058] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:37.425118 10103 raft_consensus.cc:2466] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 956ff058e6d64579a1963aabdb6d466b in term 2.
I20250627 13:54:37.431512 8189 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "14dba72d45694defb83f41c1572722ca"
dest_uuid: "e4c831df4da841929d13d6c783099b0e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37338
I20250627 13:54:37.431707 8189 raft_consensus.cc:604] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Received request to transfer leadership
I20250627 13:54:37.431816 8189 raft_consensus.cc:612] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250627 13:54:37.438128 9709 consensus_queue.cc:237] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 2.24, Last appended by leader: 16, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } }
W20250627 13:54:37.440680 9647 consensus_peers.cc:487] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a -> Peer e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): Couldn't send request to peer e4c831df4da841929d13d6c783099b0e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:37.440657 9973 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 24. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20250627 13:54:37.441232 14961 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20250627 13:54:37.442580 7801 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 24. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20250627 13:54:37.443207 15288 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20250627 13:54:37.444176 15289 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 LEADER]: Committing config change with OpId 2.25: config changed from index 24 to 25, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
I20250627 13:54:37.444823 9973 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Committing config change with OpId 2.25: config changed from index 24 to 25, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
I20250627 13:54:37.446430 7801 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Committing config change with OpId 2.25: config changed from index 24 to 25, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
I20250627 13:54:37.453006 7661 catalog_manager.cc:5582] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 24 to 25, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
I20250627 13:54:37.478539 8063 consensus_queue.cc:237] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } }
I20250627 13:54:37.480597 9840 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20250627 13:54:37.480577 7801 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
W20250627 13:54:37.480683 7999 consensus_peers.cc:487] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 -> Peer e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): Couldn't send request to peer e4c831df4da841929d13d6c783099b0e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:37.481082 15182 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20250627 13:54:37.481324 15181 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20250627 13:54:37.483424 15181 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index 24 to 25, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
I20250627 13:54:37.484170 7801 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index 24 to 25, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
I20250627 13:54:37.484269 9840 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index 24 to 25, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } }
I20250627 13:54:37.488092 7661 catalog_manager.cc:5582] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 reported cstate change: config changed from index 24 to 25, NON_VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:37.771929 8063 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Refusing update from remote peer 956ff058e6d64579a1963aabdb6d466b: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 2 index: 17. (index mismatch)
I20250627 13:54:37.772439 14808 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250627 13:54:37.811769 10103 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Refusing update from remote peer 956ff058e6d64579a1963aabdb6d466b: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 2 index: 17. (index mismatch)
I20250627 13:54:37.812253 14808 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250627 13:54:37.816944 8189 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Refusing update from remote peer 956ff058e6d64579a1963aabdb6d466b: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 2 index: 17. (index mismatch)
I20250627 13:54:37.818122 14808 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250627 13:54:37.822454 9840 consensus_queue.cc:237] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 2.17, Last appended by leader: 16, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:37.824117 8063 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Refusing update from remote peer 956ff058e6d64579a1963aabdb6d466b: Log matching property violated. Preceding OpId in replica: term: 2 index: 17. Preceding OpId from leader: term: 2 index: 18. (index mismatch)
I20250627 13:54:37.824277 10103 raft_consensus.cc:1273] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Refusing update from remote peer 956ff058e6d64579a1963aabdb6d466b: Log matching property violated. Preceding OpId in replica: term: 2 index: 17. Preceding OpId from leader: term: 2 index: 18. (index mismatch)
I20250627 13:54:37.824784 14808 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250627 13:54:37.825011 15171 consensus_queue.cc:1035] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250627 13:54:37.827378 14808 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P 956ff058e6d64579a1963aabdb6d466b [term 2 LEADER]: Committing config change with OpId 2.18: config changed from index 16 to 18, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:37.827857 10103 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Committing config change with OpId 2.18: config changed from index 16 to 18, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:37.828751 8063 raft_consensus.cc:2953] T 14dba72d45694defb83f41c1572722ca P 9cf479bdcd0242738d2a77db4ad9f660 [term 2 FOLLOWER]: Committing config change with OpId 2.18: config changed from index 16 to 18, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:37.833339 7661 catalog_manager.cc:5582] T 14dba72d45694defb83f41c1572722ca P dede81a8c0164427a61532d01b6fc157 reported cstate change: config changed from index 16 to 18, VOTER e4c831df4da841929d13d6c783099b0e (127.0.103.68) evicted. New cstate: current_term: 2 leader_uuid: "956ff058e6d64579a1963aabdb6d466b" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:37.833918 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 14dba72d45694defb83f41c1572722ca with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:37.836951 8171 tablet_service.cc:1515] Processing DeleteTablet for tablet 14dba72d45694defb83f41c1572722ca with delete_type TABLET_DATA_TOMBSTONED (TS e4c831df4da841929d13d6c783099b0e not found in new config with opid_index 18) from {username='slave'} at 127.0.0.1:48616
I20250627 13:54:37.837638 15350 tablet_replica.cc:331] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e: stopping tablet replica
I20250627 13:54:37.837940 15350 raft_consensus.cc:2241] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:37.838130 15350 raft_consensus.cc:2270] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:37.838984 15350 ts_tablet_manager.cc:1905] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:37.843451 15350 ts_tablet_manager.cc:1918] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.17
I20250627 13:54:37.843600 15350 log.cc:1199] T 14dba72d45694defb83f41c1572722ca P e4c831df4da841929d13d6c783099b0e: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-3/wal/wals/14dba72d45694defb83f41c1572722ca
I20250627 13:54:37.844327 7648 catalog_manager.cc:4928] TS e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139): tablet 14dba72d45694defb83f41c1572722ca (table rebalance_test_table_6 [id=f83cab74056a4e7681a701169d1c1716]) successfully deleted
I20250627 13:54:37.900712 15368 ts_tablet_manager.cc:927] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: Initiating tablet copy from peer 9cf479bdcd0242738d2a77db4ad9f660 (127.0.103.67:38933)
I20250627 13:54:37.903146 15368 tablet_copy_client.cc:323] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38933
I20250627 13:54:37.903798 8082 tablet_copy_service.cc:140] P 9cf479bdcd0242738d2a77db4ad9f660: Received BeginTabletCopySession request for tablet 644985e3086d4614b6598a727cb60c1d from peer e4c831df4da841929d13d6c783099b0e ({username='slave'} at 127.0.103.68:59937)
I20250627 13:54:37.903972 8082 tablet_copy_service.cc:161] P 9cf479bdcd0242738d2a77db4ad9f660: Beginning new tablet copy session on tablet 644985e3086d4614b6598a727cb60c1d from peer e4c831df4da841929d13d6c783099b0e at {username='slave'} at 127.0.103.68:59937: session id = e4c831df4da841929d13d6c783099b0e-644985e3086d4614b6598a727cb60c1d
I20250627 13:54:37.906755 8082 tablet_copy_source_session.cc:215] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:37.907971 15368 ts_tablet_manager.cc:1905] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250627 13:54:37.915771 15368 ts_tablet_manager.cc:1918] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.23
I20250627 13:54:37.916030 15368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 644985e3086d4614b6598a727cb60c1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:37.918198 15368 tablet_copy_client.cc:806] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:37.918411 15368 tablet_copy_client.cc:670] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:37.920095 15368 tablet_copy_client.cc:538] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:37.923076 15368 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
I20250627 13:54:38.018199 15368 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=25 overwritten=0 applied=25 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:38.018646 15368 tablet_bootstrap.cc:492] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: Bootstrap complete.
I20250627 13:54:38.018949 15368 ts_tablet_manager.cc:1397] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.096s user 0.076s sys 0.008s
I20250627 13:54:38.019784 15368 raft_consensus.cc:357] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } }
I20250627 13:54:38.020037 15368 raft_consensus.cc:738] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: LEARNER
I20250627 13:54:38.020275 15368 consensus_queue.cc:260] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } }
I20250627 13:54:38.020864 15368 ts_tablet_manager.cc:1428] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250627 13:54:38.021534 8082 tablet_copy_service.cc:342] P 9cf479bdcd0242738d2a77db4ad9f660: Request end of tablet copy session e4c831df4da841929d13d6c783099b0e-644985e3086d4614b6598a727cb60c1d received from {username='slave'} at 127.0.103.68:59937
I20250627 13:54:38.021692 8082 tablet_copy_service.cc:434] P 9cf479bdcd0242738d2a77db4ad9f660: ending tablet copy session e4c831df4da841929d13d6c783099b0e-644985e3086d4614b6598a727cb60c1d on tablet 644985e3086d4614b6598a727cb60c1d with peer e4c831df4da841929d13d6c783099b0e
I20250627 13:54:38.037829 15368 ts_tablet_manager.cc:927] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: Initiating tablet copy from peer 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101)
I20250627 13:54:38.040423 15368 tablet_copy_client.cc:323] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.69:33101
I20250627 13:54:38.041160 9731 tablet_copy_service.cc:140] P 2a1282ebe92a4c9a92eed3b13f69012a: Received BeginTabletCopySession request for tablet e3053b529a264e8697a1df9e17eeea81 from peer e4c831df4da841929d13d6c783099b0e ({username='slave'} at 127.0.103.68:51963)
I20250627 13:54:38.041337 9731 tablet_copy_service.cc:161] P 2a1282ebe92a4c9a92eed3b13f69012a: Beginning new tablet copy session on tablet e3053b529a264e8697a1df9e17eeea81 from peer e4c831df4da841929d13d6c783099b0e at {username='slave'} at 127.0.103.68:51963: session id = e4c831df4da841929d13d6c783099b0e-e3053b529a264e8697a1df9e17eeea81
I20250627 13:54:38.044198 9731 tablet_copy_source_session.cc:215] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:38.045439 15368 ts_tablet_manager.cc:1905] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250627 13:54:38.051715 15368 ts_tablet_manager.cc:1918] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.23
I20250627 13:54:38.051980 15368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3053b529a264e8697a1df9e17eeea81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:38.054404 15368 tablet_copy_client.cc:806] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:38.054589 15368 tablet_copy_client.cc:670] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:38.056535 15368 tablet_copy_client.cc:538] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:38.059465 15368 tablet_bootstrap.cc:492] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: Bootstrap starting.
I20250627 13:54:38.077008 9691 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:38.094002 7782 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:38.101188 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:38.125581 9822 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:38.126658 8041 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:38.188374 10083 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:38.189946 15368 tablet_bootstrap.cc:492] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=25 overwritten=0 applied=25 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:38.190353 15368 tablet_bootstrap.cc:492] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: Bootstrap complete.
I20250627 13:54:38.190608 15368 ts_tablet_manager.cc:1397] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: Time spent bootstrapping tablet: real 0.131s user 0.085s sys 0.000s
I20250627 13:54:38.191308 15368 raft_consensus.cc:357] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } }
I20250627 13:54:38.191512 15368 raft_consensus.cc:738] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e4c831df4da841929d13d6c783099b0e, State: Initialized, Role: LEARNER
I20250627 13:54:38.191713 15368 consensus_queue.cc:260] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 25, Last appended: 2.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } }
I20250627 13:54:38.192174 15368 ts_tablet_manager.cc:1428] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250627 13:54:38.192906 9731 tablet_copy_service.cc:342] P 2a1282ebe92a4c9a92eed3b13f69012a: Request end of tablet copy session e4c831df4da841929d13d6c783099b0e-e3053b529a264e8697a1df9e17eeea81 received from {username='slave'} at 127.0.103.68:51963
I20250627 13:54:38.193044 9731 tablet_copy_service.cc:434] P 2a1282ebe92a4c9a92eed3b13f69012a: ending tablet copy session e4c831df4da841929d13d6c783099b0e-e3053b529a264e8697a1df9e17eeea81 on tablet e3053b529a264e8697a1df9e17eeea81 with peer e4c831df4da841929d13d6c783099b0e
I20250627 13:54:38.216971 8171 tablet_service.cc:1430] Tablet server has 3 leaders and 0 scanners
I20250627 13:54:38.230998 7911 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:38.353163 8189 consensus_queue.cc:237] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } }
I20250627 13:54:38.355662 10103 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
W20250627 13:54:38.355792 8128 consensus_peers.cc:487] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e -> Peer a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675): Couldn't send request to peer a41a89ae8ed743edb2b5cf4490d8c854. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:38.356205 15174 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250627 13:54:38.357194 9840 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250627 13:54:38.357990 15447 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250627 13:54:38.359365 15174 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:38.359869 10103 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:38.365756 7659 catalog_manager.cc:5582] T 03640a5124d2405faf4a26d0e22e1ffc P dede81a8c0164427a61532d01b6fc157 reported cstate change: config changed from index 13 to 14, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New cstate: current_term: 2 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:38.400246 9840 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } } }
I20250627 13:54:38.438472 8189 raft_consensus.cc:1215] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.24->[1.25-1.25] Dedup: 1.25->[]
I20250627 13:54:38.613456 8189 raft_consensus.cc:1215] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.24->[2.25-2.25] Dedup: 2.25->[]
I20250627 13:54:38.897734 15514 ts_tablet_manager.cc:927] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854: Initiating tablet copy from peer e4c831df4da841929d13d6c783099b0e (127.0.103.68:46139)
I20250627 13:54:38.898243 15514 tablet_copy_client.cc:323] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.68:46139
I20250627 13:54:38.898833 8213 tablet_copy_service.cc:140] P e4c831df4da841929d13d6c783099b0e: Received BeginTabletCopySession request for tablet 03640a5124d2405faf4a26d0e22e1ffc from peer a41a89ae8ed743edb2b5cf4490d8c854 ({username='slave'} at 127.0.103.66:33401)
I20250627 13:54:38.898979 8213 tablet_copy_service.cc:161] P e4c831df4da841929d13d6c783099b0e: Beginning new tablet copy session on tablet 03640a5124d2405faf4a26d0e22e1ffc from peer a41a89ae8ed743edb2b5cf4490d8c854 at {username='slave'} at 127.0.103.66:33401: session id = a41a89ae8ed743edb2b5cf4490d8c854-03640a5124d2405faf4a26d0e22e1ffc
I20250627 13:54:38.901465 8213 tablet_copy_source_session.cc:215] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:38.902585 15514 ts_tablet_manager.cc:1905] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250627 13:54:38.908658 15514 ts_tablet_manager.cc:1918] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.12
I20250627 13:54:38.909004 15514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03640a5124d2405faf4a26d0e22e1ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:38.911911 15514 tablet_copy_client.cc:806] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:38.912137 15514 tablet_copy_client.cc:670] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:38.913702 15514 tablet_copy_client.cc:538] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:38.916128 15514 tablet_bootstrap.cc:492] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap starting.
I20250627 13:54:38.934033 15460 raft_consensus.cc:1062] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660: attempting to promote NON_VOTER e4c831df4da841929d13d6c783099b0e to VOTER
I20250627 13:54:38.934893 15460 consensus_queue.cc:237] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } }
I20250627 13:54:38.937510 9840 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250627 13:54:38.938206 8189 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 LEARNER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250627 13:54:38.939810 14624 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250627 13:54:38.940404 15528 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250627 13:54:38.940837 7801 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250627 13:54:38.942471 15528 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250627 13:54:38.944857 14624 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index 25 to 26, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:38.945446 7801 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:38.950578 8189 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:38.951004 7659 catalog_manager.cc:5582] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 25 to 26, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:38.963935 8063 consensus_queue.cc:237] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } }
I20250627 13:54:38.964802 15514 tablet_bootstrap.cc:492] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:38.965209 15514 tablet_bootstrap.cc:492] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854: Bootstrap complete.
I20250627 13:54:38.965489 15514 ts_tablet_manager.cc:1397] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854: Time spent bootstrapping tablet: real 0.049s user 0.049s sys 0.001s
I20250627 13:54:38.965858 8189 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250627 13:54:38.965950 7801 raft_consensus.cc:1273] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Refusing update from remote peer 9cf479bdcd0242738d2a77db4ad9f660: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250627 13:54:38.966439 15460 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250627 13:54:38.966449 15514 raft_consensus.cc:357] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } }
I20250627 13:54:38.966698 15514 raft_consensus.cc:738] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: a41a89ae8ed743edb2b5cf4490d8c854, State: Initialized, Role: LEARNER
I20250627 13:54:38.966857 15460 consensus_queue.cc:1035] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250627 13:54:38.966873 15514 consensus_queue.cc:260] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: NON_VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: true } }
I20250627 13:54:38.967475 15514 ts_tablet_manager.cc:1428] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:54:38.968564 8213 tablet_copy_service.cc:342] P e4c831df4da841929d13d6c783099b0e: Request end of tablet copy session a41a89ae8ed743edb2b5cf4490d8c854-03640a5124d2405faf4a26d0e22e1ffc received from {username='slave'} at 127.0.103.66:33401
I20250627 13:54:38.968683 8213 tablet_copy_service.cc:434] P e4c831df4da841929d13d6c783099b0e: ending tablet copy session a41a89ae8ed743edb2b5cf4490d8c854-03640a5124d2405faf4a26d0e22e1ffc on tablet 03640a5124d2405faf4a26d0e22e1ffc with peer a41a89ae8ed743edb2b5cf4490d8c854
I20250627 13:54:38.969718 15528 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 9cf479bdcd0242738d2a77db4ad9f660 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:38.970253 7801 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:38.975733 7645 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 644985e3086d4614b6598a727cb60c1d with cas_config_opid_index 26: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:38.975735 8189 raft_consensus.cc:2953] T 644985e3086d4614b6598a727cb60c1d P e4c831df4da841929d13d6c783099b0e [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:38.977064 7653 catalog_manager.cc:5582] T 644985e3086d4614b6598a727cb60c1d P 83574c1bf8ea45adaff939158f93c096 reported cstate change: config changed from index 26 to 27, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New cstate: current_term: 1 leader_uuid: "9cf479bdcd0242738d2a77db4ad9f660" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:38.980410 9822 tablet_service.cc:1515] Processing DeleteTablet for tablet 644985e3086d4614b6598a727cb60c1d with delete_type TABLET_DATA_TOMBSTONED (TS 956ff058e6d64579a1963aabdb6d466b not found in new config with opid_index 27) from {username='slave'} at 127.0.0.1:55228
I20250627 13:54:38.981117 15536 tablet_replica.cc:331] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: stopping tablet replica
I20250627 13:54:38.981303 15536 raft_consensus.cc:2241] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:38.981583 15536 pending_rounds.cc:70] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: Trying to abort 1 pending ops.
I20250627 13:54:38.981722 15536 pending_rounds.cc:77] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: Aborting op as it isn't in flight: id { term: 1 index: 26 } timestamp: 7172229033714896896 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "644985e3086d4614b6598a727cb60c1d" old_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: true } } } new_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } } }
I20250627 13:54:38.982255 15536 raft_consensus.cc:2915] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Aborting config change with OpId 1.26: Aborted: Op aborted
I20250627 13:54:38.982419 15536 raft_consensus.cc:2887] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250627 13:54:38.982578 15536 raft_consensus.cc:2270] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:38.985024 15536 ts_tablet_manager.cc:1905] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:38.990674 15536 ts_tablet_manager.cc:1918] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.26
I20250627 13:54:38.990815 15536 log.cc:1199] T 644985e3086d4614b6598a727cb60c1d P 956ff058e6d64579a1963aabdb6d466b: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal/wals/644985e3086d4614b6598a727cb60c1d
I20250627 13:54:38.991641 7645 catalog_manager.cc:4928] TS 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): tablet 644985e3086d4614b6598a727cb60c1d (table rebalance_test_table_1 [id=8961aa0df2f1488f82cd55cfa22ad214]) successfully deleted
I20250627 13:54:39.055574 15483 raft_consensus.cc:1062] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a: attempting to promote NON_VOTER e4c831df4da841929d13d6c783099b0e to VOTER
I20250627 13:54:39.056267 15483 consensus_queue.cc:237] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 2.25, Last appended by leader: 16, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } }
I20250627 13:54:39.058588 8189 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 2 LEARNER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20250627 13:54:39.058635 7801 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20250627 13:54:39.058662 9973 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20250627 13:54:39.059175 15483 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250627 13:54:39.059603 15558 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250627 13:54:39.059962 14961 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250627 13:54:39.062825 15558 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 LEADER]: Committing config change with OpId 2.26: config changed from index 25 to 26, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:39.063485 7801 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Committing config change with OpId 2.26: config changed from index 25 to 26, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:39.067256 8189 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Committing config change with OpId 2.26: config changed from index 25 to 26, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:39.068704 7659 catalog_manager.cc:5582] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: config changed from index 25 to 26, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:39.076939 9973 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Committing config change with OpId 2.26: config changed from index 25 to 26, e4c831df4da841929d13d6c783099b0e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:39.079931 9709 consensus_queue.cc:237] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 16, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } }
I20250627 13:54:39.081543 8189 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20250627 13:54:39.082058 15483 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250627 13:54:39.083992 9974 raft_consensus.cc:1273] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20250627 13:54:39.084043 15483 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index 26 to 27, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:39.087596 7648 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet e3053b529a264e8697a1df9e17eeea81 with cas_config_opid_index 26: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:39.087611 15482 consensus_queue.cc:1035] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250627 13:54:39.088376 8189 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P e4c831df4da841929d13d6c783099b0e [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index 26 to 27, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:39.088949 7661 catalog_manager.cc:5582] T e3053b529a264e8697a1df9e17eeea81 P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: config changed from index 26 to 27, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New cstate: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:39.092041 7782 tablet_service.cc:1515] Processing DeleteTablet for tablet e3053b529a264e8697a1df9e17eeea81 with delete_type TABLET_DATA_TOMBSTONED (TS 83574c1bf8ea45adaff939158f93c096 not found in new config with opid_index 27) from {username='slave'} at 127.0.0.1:45826
I20250627 13:54:39.092028 9973 raft_consensus.cc:2953] T e3053b529a264e8697a1df9e17eeea81 P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index 26 to 27, VOTER 83574c1bf8ea45adaff939158f93c096 (127.0.103.65) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } }
I20250627 13:54:39.092592 15564 tablet_replica.cc:331] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096: stopping tablet replica
I20250627 13:54:39.092788 15564 raft_consensus.cc:2241] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:39.093106 15564 raft_consensus.cc:2270] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:39.094077 15564 ts_tablet_manager.cc:1905] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:39.098932 15564 ts_tablet_manager.cc:1918] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.26
I20250627 13:54:39.099038 15564 log.cc:1199] T e3053b529a264e8697a1df9e17eeea81 P 83574c1bf8ea45adaff939158f93c096: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-0/wal/wals/e3053b529a264e8697a1df9e17eeea81
I20250627 13:54:39.099635 7645 catalog_manager.cc:4928] TS 83574c1bf8ea45adaff939158f93c096 (127.0.103.65:37467): tablet e3053b529a264e8697a1df9e17eeea81 (table rebalance_test_table_0 [id=d8bfc9268a4a4c2c92bc20d56ac73086]) successfully deleted
I20250627 13:54:39.339576 7927 raft_consensus.cc:1215] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.13->[2.14-2.14] Dedup: 2.14->[]
I20250627 13:54:39.415684 9691 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:39.468861 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:54:39.491303 7648 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet e2bd060a4a9e4765b45b0242ae56e489 with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER failed with leader 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101) due to error UNKNOWN_ERROR; will retry: Invalid argument: Cannot remove peer 2a1282ebe92a4c9a92eed3b13f69012a from the config because it is the leader. Force another leader to be elected to remove this peer. Consensus state: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:39.504290 8171 tablet_service.cc:1430] Tablet server has 3 leaders and 0 scanners
I20250627 13:54:39.507633 7782 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:39.509472 10083 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:39.514313 9822 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:39.521982 7911 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:39.569684 8041 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:39.804651 9691 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:39.805049 7782 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:39.807919 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:39.829483 15452 raft_consensus.cc:1062] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e: attempting to promote NON_VOTER a41a89ae8ed743edb2b5cf4490d8c854 to VOTER
I20250627 13:54:39.830339 15452 consensus_queue.cc:237] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } }
I20250627 13:54:39.840276 7911 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:39.867022 7927 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 LEARNER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20250627 13:54:39.868893 15612 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250627 13:54:39.870325 10083 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:39.902621 10103 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20250627 13:54:39.903213 15452 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250627 13:54:39.919087 9822 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:39.922730 9840 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20250627 13:54:39.924408 15612 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 2 LEADER]: Committing config change with OpId 2.15: config changed from index 14 to 15, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:39.932514 10103 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:39.936578 7661 catalog_manager.cc:5582] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 14 to 15, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:39.937153 7927 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:39.944206 15452 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250627 13:54:39.946364 8041 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:39.961581 8171 tablet_service.cc:1430] Tablet server has 3 leaders and 0 scanners
I20250627 13:54:39.979722 9840 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false replace: true } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:40.034759 8189 consensus_queue.cc:237] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 2.15, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } }
I20250627 13:54:40.036715 7927 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20250627 13:54:40.037019 10103 raft_consensus.cc:1273] T 03640a5124d2405faf4a26d0e22e1ffc P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Refusing update from remote peer e4c831df4da841929d13d6c783099b0e: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20250627 13:54:40.037567 15452 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250627 13:54:40.038730 15452 consensus_queue.cc:1035] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250627 13:54:40.040917 15612 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e [term 2 LEADER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:40.041538 10103 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P dede81a8c0164427a61532d01b6fc157 [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:40.043804 7648 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 03640a5124d2405faf4a26d0e22e1ffc with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:40.045214 7653 catalog_manager.cc:5582] T 03640a5124d2405faf4a26d0e22e1ffc P e4c831df4da841929d13d6c783099b0e reported cstate change: config changed from index 15 to 16, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New cstate: current_term: 2 leader_uuid: "e4c831df4da841929d13d6c783099b0e" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:40.048878 9822 tablet_service.cc:1515] Processing DeleteTablet for tablet 03640a5124d2405faf4a26d0e22e1ffc with delete_type TABLET_DATA_TOMBSTONED (TS 956ff058e6d64579a1963aabdb6d466b not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:55228
I20250627 13:54:40.049458 15718 tablet_replica.cc:331] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b: stopping tablet replica
I20250627 13:54:40.049682 15718 raft_consensus.cc:2241] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:40.049893 15718 raft_consensus.cc:2270] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:40.050791 15718 ts_tablet_manager.cc:1905] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:40.050959 7927 raft_consensus.cc:2953] T 03640a5124d2405faf4a26d0e22e1ffc P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } } peers { permanent_uuid: "dede81a8c0164427a61532d01b6fc157" member_type: VOTER last_known_addr { host: "127.0.103.72" port: 35213 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false } } }
I20250627 13:54:40.055831 15718 ts_tablet_manager.cc:1918] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.15
I20250627 13:54:40.055989 15718 log.cc:1199] T 03640a5124d2405faf4a26d0e22e1ffc P 956ff058e6d64579a1963aabdb6d466b: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-5/wal/wals/03640a5124d2405faf4a26d0e22e1ffc
I20250627 13:54:40.056614 7645 catalog_manager.cc:4928] TS 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): tablet 03640a5124d2405faf4a26d0e22e1ffc (table rebalance_test_table_1 [id=8961aa0df2f1488f82cd55cfa22ad214]) successfully deleted
I20250627 13:54:40.326982 9691 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:40.329972 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:40.339982 9822 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:40.350191 8041 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:40.374024 7911 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:40.397171 7782 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:40.398780 10083 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:40.426353 8171 tablet_service.cc:1430] Tablet server has 3 leaders and 0 scanners
I20250627 13:54:40.525148 9708 consensus_queue.cc:237] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:40.527539 7927 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250627 13:54:40.527951 9973 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250627 13:54:40.528050 15482 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20250627 13:54:40.528573 9644 consensus_peers.cc:487] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a -> Peer 956ff058e6d64579a1963aabdb6d466b (127.0.103.70:40647): Couldn't send request to peer 956ff058e6d64579a1963aabdb6d466b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87726824b05a4b968386f57d3a7a4aad. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:54:40.528797 15482 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250627 13:54:40.530748 15787 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:40.531348 7927 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:40.535312 9973 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:40.537892 7661 catalog_manager.cc:5582] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index 13 to 14, NON_VOTER 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) added. New cstate: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } } }
I20250627 13:54:41.151046 15841 ts_tablet_manager.cc:927] T 87726824b05a4b968386f57d3a7a4aad P 956ff058e6d64579a1963aabdb6d466b: Initiating tablet copy from peer 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101)
I20250627 13:54:41.151551 15841 tablet_copy_client.cc:323] T 87726824b05a4b968386f57d3a7a4aad P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.69:33101
I20250627 13:54:41.152139 9731 tablet_copy_service.cc:140] P 2a1282ebe92a4c9a92eed3b13f69012a: Received BeginTabletCopySession request for tablet 87726824b05a4b968386f57d3a7a4aad from peer 956ff058e6d64579a1963aabdb6d466b ({username='slave'} at 127.0.103.70:49013)
I20250627 13:54:41.152292 9731 tablet_copy_service.cc:161] P 2a1282ebe92a4c9a92eed3b13f69012a: Beginning new tablet copy session on tablet 87726824b05a4b968386f57d3a7a4aad from peer 956ff058e6d64579a1963aabdb6d466b at {username='slave'} at 127.0.103.70:49013: session id = 956ff058e6d64579a1963aabdb6d466b-87726824b05a4b968386f57d3a7a4aad
I20250627 13:54:41.154394 9731 tablet_copy_source_session.cc:215] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:54:41.155491 15841 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87726824b05a4b968386f57d3a7a4aad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:41.159417 15841 tablet_copy_client.cc:806] T 87726824b05a4b968386f57d3a7a4aad P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 0 data blocks...
I20250627 13:54:41.159637 15841 tablet_copy_client.cc:670] T 87726824b05a4b968386f57d3a7a4aad P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Starting download of 1 WAL segments...
I20250627 13:54:41.161000 15841 tablet_copy_client.cc:538] T 87726824b05a4b968386f57d3a7a4aad P 956ff058e6d64579a1963aabdb6d466b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:54:41.163194 15841 tablet_bootstrap.cc:492] T 87726824b05a4b968386f57d3a7a4aad P 956ff058e6d64579a1963aabdb6d466b: Bootstrap starting.
I20250627 13:54:41.210023 15841 tablet_bootstrap.cc:492] T 87726824b05a4b968386f57d3a7a4aad P 956ff058e6d64579a1963aabdb6d466b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:54:41.210520 15841 tablet_bootstrap.cc:492] T 87726824b05a4b968386f57d3a7a4aad P 956ff058e6d64579a1963aabdb6d466b: Bootstrap complete.
I20250627 13:54:41.210815 15841 ts_tablet_manager.cc:1397] T 87726824b05a4b968386f57d3a7a4aad P 956ff058e6d64579a1963aabdb6d466b: Time spent bootstrapping tablet: real 0.048s user 0.040s sys 0.009s
I20250627 13:54:41.211539 15841 raft_consensus.cc:357] T 87726824b05a4b968386f57d3a7a4aad P 956ff058e6d64579a1963aabdb6d466b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:41.211797 15841 raft_consensus.cc:738] T 87726824b05a4b968386f57d3a7a4aad P 956ff058e6d64579a1963aabdb6d466b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 956ff058e6d64579a1963aabdb6d466b, State: Initialized, Role: LEARNER
I20250627 13:54:41.212023 15841 consensus_queue.cc:260] T 87726824b05a4b968386f57d3a7a4aad P 956ff058e6d64579a1963aabdb6d466b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: true } }
I20250627 13:54:41.212563 15841 ts_tablet_manager.cc:1428] T 87726824b05a4b968386f57d3a7a4aad P 956ff058e6d64579a1963aabdb6d466b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:54:41.213508 9731 tablet_copy_service.cc:342] P 2a1282ebe92a4c9a92eed3b13f69012a: Request end of tablet copy session 956ff058e6d64579a1963aabdb6d466b-87726824b05a4b968386f57d3a7a4aad received from {username='slave'} at 127.0.103.70:49013
I20250627 13:54:41.213675 9731 tablet_copy_service.cc:434] P 2a1282ebe92a4c9a92eed3b13f69012a: ending tablet copy session 956ff058e6d64579a1963aabdb6d466b-87726824b05a4b968386f57d3a7a4aad on tablet 87726824b05a4b968386f57d3a7a4aad with peer 956ff058e6d64579a1963aabdb6d466b
I20250627 13:54:41.725574 9842 raft_consensus.cc:1215] T 87726824b05a4b968386f57d3a7a4aad P 956ff058e6d64579a1963aabdb6d466b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.13->[2.14-2.14] Dedup: 2.14->[]
I20250627 13:54:42.146826 15840 raft_consensus.cc:1062] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a: attempting to promote NON_VOTER 956ff058e6d64579a1963aabdb6d466b to VOTER
I20250627 13:54:42.147490 15840 consensus_queue.cc:237] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:42.149878 9842 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P 956ff058e6d64579a1963aabdb6d466b [term 2 LEARNER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20250627 13:54:42.149967 7927 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20250627 13:54:42.150171 9973 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20250627 13:54:42.150470 15890 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250627 13:54:42.150718 15923 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250627 13:54:42.151012 15840 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250627 13:54:42.155193 7927 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:42.154433 15923 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 LEADER]: Committing config change with OpId 2.15: config changed from index 14 to 15, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:42.160228 9973 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:42.161741 7658 catalog_manager.cc:5582] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 reported cstate change: config changed from index 14 to 15, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:42.166591 9842 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, 956ff058e6d64579a1963aabdb6d466b (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "a41a89ae8ed743edb2b5cf4490d8c854" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43675 } attrs { promote: false replace: true } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:42.177937 9708 consensus_queue.cc:237] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 2.15, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } }
I20250627 13:54:42.179659 9842 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20250627 13:54:42.179996 9973 raft_consensus.cc:1273] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Refusing update from remote peer 2a1282ebe92a4c9a92eed3b13f69012a: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20250627 13:54:42.180197 15840 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250627 13:54:42.181394 15840 consensus_queue.cc:1035] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250627 13:54:42.182915 15890 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a [term 2 LEADER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:42.183562 9842 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 956ff058e6d64579a1963aabdb6d466b [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:42.185786 7648 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 87726824b05a4b968386f57d3a7a4aad with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250627 13:54:42.186698 7661 catalog_manager.cc:5582] T 87726824b05a4b968386f57d3a7a4aad P 2a1282ebe92a4c9a92eed3b13f69012a reported cstate change: config changed from index 15 to 16, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New cstate: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250627 13:54:42.188334 9974 raft_consensus.cc:2953] T 87726824b05a4b968386f57d3a7a4aad P 6686c6c365bf4f4582058890161676fe [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "6686c6c365bf4f4582058890161676fe" member_type: VOTER last_known_addr { host: "127.0.103.71" port: 37859 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:54:42.192497 7911 tablet_service.cc:1515] Processing DeleteTablet for tablet 87726824b05a4b968386f57d3a7a4aad with delete_type TABLET_DATA_TOMBSTONED (TS a41a89ae8ed743edb2b5cf4490d8c854 not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:59302
I20250627 13:54:42.198066 15932 tablet_replica.cc:331] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854: stopping tablet replica
I20250627 13:54:42.198246 15932 raft_consensus.cc:2241] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250627 13:54:42.198433 15932 raft_consensus.cc:2270] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250627 13:54:42.199155 15932 ts_tablet_manager.cc:1905] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250627 13:54:42.203459 15932 ts_tablet_manager.cc:1918] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.15
I20250627 13:54:42.203543 15932 log.cc:1199] T 87726824b05a4b968386f57d3a7a4aad P a41a89ae8ed743edb2b5cf4490d8c854: Deleting WAL directory at /tmp/dist-test-taskNzQG6V/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1751032333768462-413-0/raft_consensus-itest-cluster/ts-1/wal/wals/87726824b05a4b968386f57d3a7a4aad
I20250627 13:54:42.204111 7647 catalog_manager.cc:4928] TS a41a89ae8ed743edb2b5cf4490d8c854 (127.0.103.66:43675): tablet 87726824b05a4b968386f57d3a7a4aad (table rebalance_test_table_1 [id=8961aa0df2f1488f82cd55cfa22ad214]) successfully deleted
I20250627 13:54:42.499651 7911 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:42.516237 9691 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:42.519155 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:42.533903 10083 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:42.552253 8171 tablet_service.cc:1430] Tablet server has 3 leaders and 0 scanners
I20250627 13:54:42.553277 9822 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:42.554842 7782 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:42.602432 8041 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:42.830135 7782 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:42.873181 8041 tablet_service.cc:1430] Tablet server has 7 leaders and 0 scanners
I20250627 13:54:42.891702 9822 tablet_service.cc:1430] Tablet server has 4 leaders and 0 scanners
I20250627 13:54:42.893996 10083 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:42.894903 7911 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:42.896647 9691 tablet_service.cc:1430] Tablet server has 5 leaders and 0 scanners
I20250627 13:54:42.909355 9954 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:42.914158 8171 tablet_service.cc:1430] Tablet server has 3 leaders and 0 scanners
W20250627 13:54:43.601778 7648 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet e2bd060a4a9e4765b45b0242ae56e489 with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER failed with leader 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101) due to error UNKNOWN_ERROR; will retry: Invalid argument: Cannot remove peer 2a1282ebe92a4c9a92eed3b13f69012a from the config because it is the leader. Force another leader to be elected to remove this peer. Consensus state: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
W20250627 13:54:51.823273 7648 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet e2bd060a4a9e4765b45b0242ae56e489 with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER failed with leader 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101) due to error UNKNOWN_ERROR; will retry: Invalid argument: Cannot remove peer 2a1282ebe92a4c9a92eed3b13f69012a from the config because it is the leader. Force another leader to be elected to remove this peer. Consensus state: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
W20250627 13:55:08.249430 7648 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet e2bd060a4a9e4765b45b0242ae56e489 with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER failed with leader 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101) due to error UNKNOWN_ERROR; will retry: Invalid argument: Cannot remove peer 2a1282ebe92a4c9a92eed3b13f69012a from the config because it is the leader. Force another leader to be elected to remove this peer. Consensus state: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:55:40.521517 7650 maintenance_manager.cc:419] P e7a954e8f6cf484a984a188e218b202d: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033374
I20250627 13:55:40.552778 7649 maintenance_manager.cc:643] P e7a954e8f6cf484a984a188e218b202d: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.031s user 0.018s sys 0.009s Metrics: {"bytes_written":20141,"cfile_init":1,"dirs.queue_time_us":318,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1068,"drs_written":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4,"lbm_write_time_us":896,"lbm_writes_lt_1ms":30,"peak_mem_usage":0,"rows_written":43,"thread_start_us":388,"threads_started":2,"wal-append.queue_time_us":241}
I20250627 13:55:40.553974 7650 maintenance_manager.cc:419] P e7a954e8f6cf484a984a188e218b202d: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 14999 bytes on disk
I20250627 13:55:40.554641 7649 maintenance_manager.cc:643] P e7a954e8f6cf484a984a188e218b202d: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
W20250627 13:55:41.028800 7648 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet e2bd060a4a9e4765b45b0242ae56e489 with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER failed with leader 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101) due to error UNKNOWN_ERROR; will retry: Invalid argument: Cannot remove peer 2a1282ebe92a4c9a92eed3b13f69012a from the config because it is the leader. Force another leader to be elected to remove this peer. Consensus state: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:55:48.378517 7979 maintenance_manager.cc:419] P a41a89ae8ed743edb2b5cf4490d8c854: Scheduling FlushMRSOp(b265cfda4bfe48908659208ec8eeab2d): perf score=0.033359
I20250627 13:55:48.412988 8109 maintenance_manager.cc:419] P 9cf479bdcd0242738d2a77db4ad9f660: Scheduling FlushMRSOp(d2eb537d49804e1b8ec5a217b1564988): perf score=0.033372
I20250627 13:55:48.474821 7870 maintenance_manager.cc:643] P a41a89ae8ed743edb2b5cf4490d8c854: FlushMRSOp(b265cfda4bfe48908659208ec8eeab2d) complete. Timing: real 0.096s user 0.061s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":397,"dirs.queue_time_us":822,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":1493,"drs_written":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4,"lbm_write_time_us":621,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":13142,"threads_started":3,"wal-append.queue_time_us":12355}
I20250627 13:55:48.477351 7979 maintenance_manager.cc:419] P a41a89ae8ed743edb2b5cf4490d8c854: Scheduling UndoDeltaBlockGCOp(b265cfda4bfe48908659208ec8eeab2d): 4840 bytes on disk
I20250627 13:55:48.479131 7870 maintenance_manager.cc:643] P a41a89ae8ed743edb2b5cf4490d8c854: UndoDeltaBlockGCOp(b265cfda4bfe48908659208ec8eeab2d) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":156,"lbm_reads_lt_1ms":4}
I20250627 13:55:48.535727 8000 maintenance_manager.cc:643] P 9cf479bdcd0242738d2a77db4ad9f660: FlushMRSOp(d2eb537d49804e1b8ec5a217b1564988) complete. Timing: real 0.122s user 0.068s sys 0.027s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":12505,"dirs.queue_time_us":2860,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":1767,"drs_written":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4,"lbm_write_time_us":880,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":15290,"threads_started":3,"wal-append.queue_time_us":353}
I20250627 13:55:48.537358 8109 maintenance_manager.cc:419] P 9cf479bdcd0242738d2a77db4ad9f660: Scheduling UndoDeltaBlockGCOp(d2eb537d49804e1b8ec5a217b1564988): 4585 bytes on disk
I20250627 13:55:48.538092 8000 maintenance_manager.cc:643] P 9cf479bdcd0242738d2a77db4ad9f660: UndoDeltaBlockGCOp(d2eb537d49804e1b8ec5a217b1564988) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20250627 13:55:49.277470 8239 maintenance_manager.cc:419] P e4c831df4da841929d13d6c783099b0e: Scheduling FlushMRSOp(03640a5124d2405faf4a26d0e22e1ffc): perf score=0.033334
I20250627 13:55:49.291790 8109 maintenance_manager.cc:419] P 9cf479bdcd0242738d2a77db4ad9f660: Scheduling FlushMRSOp(644985e3086d4614b6598a727cb60c1d): perf score=0.033341
I20250627 13:55:49.354898 8000 maintenance_manager.cc:643] P 9cf479bdcd0242738d2a77db4ad9f660: FlushMRSOp(644985e3086d4614b6598a727cb60c1d) complete. Timing: real 0.063s user 0.043s sys 0.014s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":8118,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":9812,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":782,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":14957,"threads_started":2,"wal-append.queue_time_us":6812}
I20250627 13:55:49.356786 8109 maintenance_manager.cc:419] P 9cf479bdcd0242738d2a77db4ad9f660: Scheduling UndoDeltaBlockGCOp(644985e3086d4614b6598a727cb60c1d): 4926 bytes on disk
I20250627 13:55:49.357976 8000 maintenance_manager.cc:643] P 9cf479bdcd0242738d2a77db4ad9f660: UndoDeltaBlockGCOp(644985e3086d4614b6598a727cb60c1d) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":156,"lbm_reads_lt_1ms":4}
I20250627 13:55:49.360332 8130 maintenance_manager.cc:643] P e4c831df4da841929d13d6c783099b0e: FlushMRSOp(03640a5124d2405faf4a26d0e22e1ffc) complete. Timing: real 0.082s user 0.057s sys 0.018s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":470,"dirs.queue_time_us":7901,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":10194,"drs_written":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4,"lbm_write_time_us":809,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":13870,"threads_started":3,"wal-append.queue_time_us":5402}
I20250627 13:55:49.362391 8239 maintenance_manager.cc:419] P e4c831df4da841929d13d6c783099b0e: Scheduling UndoDeltaBlockGCOp(03640a5124d2405faf4a26d0e22e1ffc): 4586 bytes on disk
I20250627 13:55:49.363271 8130 maintenance_manager.cc:643] P e4c831df4da841929d13d6c783099b0e: UndoDeltaBlockGCOp(03640a5124d2405faf4a26d0e22e1ffc) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20250627 13:55:50.612800 8109 maintenance_manager.cc:419] P 9cf479bdcd0242738d2a77db4ad9f660: Scheduling FlushMRSOp(5f176b66fdce4398be3492a5f3b4ea27): perf score=0.033334
I20250627 13:55:50.660897 8000 maintenance_manager.cc:643] P 9cf479bdcd0242738d2a77db4ad9f660: FlushMRSOp(5f176b66fdce4398be3492a5f3b4ea27) complete. Timing: real 0.048s user 0.042s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":4058,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":1502,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":796,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":4147,"threads_started":2,"wal-append.queue_time_us":273}
I20250627 13:55:50.661901 8109 maintenance_manager.cc:419] P 9cf479bdcd0242738d2a77db4ad9f660: Scheduling UndoDeltaBlockGCOp(5f176b66fdce4398be3492a5f3b4ea27): 4585 bytes on disk
I20250627 13:55:50.662382 8000 maintenance_manager.cc:643] P 9cf479bdcd0242738d2a77db4ad9f660: UndoDeltaBlockGCOp(5f176b66fdce4398be3492a5f3b4ea27) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20250627 13:55:50.742224 7979 maintenance_manager.cc:419] P a41a89ae8ed743edb2b5cf4490d8c854: Scheduling FlushMRSOp(da90822ff15e4e5da05114ef38d3970a): perf score=0.033369
I20250627 13:55:50.774663 7870 maintenance_manager.cc:643] P a41a89ae8ed743edb2b5cf4490d8c854: FlushMRSOp(da90822ff15e4e5da05114ef38d3970a) complete. Timing: real 0.032s user 0.031s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":315,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":1343,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":557,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":394,"threads_started":2,"wal-append.queue_time_us":260}
I20250627 13:55:50.775669 7979 maintenance_manager.cc:419] P a41a89ae8ed743edb2b5cf4490d8c854: Scheduling UndoDeltaBlockGCOp(da90822ff15e4e5da05114ef38d3970a): 4840 bytes on disk
I20250627 13:55:50.776296 7870 maintenance_manager.cc:643] P a41a89ae8ed743edb2b5cf4490d8c854: UndoDeltaBlockGCOp(da90822ff15e4e5da05114ef38d3970a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20250627 13:55:51.872352 8239 maintenance_manager.cc:419] P e4c831df4da841929d13d6c783099b0e: Scheduling FlushMRSOp(6f5e370b77cc42fdb0b886ea19af06c3): perf score=0.033346
I20250627 13:55:51.903772 8130 maintenance_manager.cc:643] P e4c831df4da841929d13d6c783099b0e: FlushMRSOp(6f5e370b77cc42fdb0b886ea19af06c3) complete. Timing: real 0.031s user 0.016s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":330,"dirs.run_cpu_time_us":231,"dirs.run_wall_time_us":1382,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":582,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":421,"threads_started":2,"wal-append.queue_time_us":296}
I20250627 13:55:51.904659 8239 maintenance_manager.cc:419] P e4c831df4da841929d13d6c783099b0e: Scheduling UndoDeltaBlockGCOp(6f5e370b77cc42fdb0b886ea19af06c3): 4840 bytes on disk
I20250627 13:55:51.905128 8130 maintenance_manager.cc:643] P e4c831df4da841929d13d6c783099b0e: UndoDeltaBlockGCOp(6f5e370b77cc42fdb0b886ea19af06c3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20250627 13:55:51.933164 7848 maintenance_manager.cc:419] P 83574c1bf8ea45adaff939158f93c096: Scheduling FlushMRSOp(5184c3f8e465472e9a98c14f8e703415): perf score=0.033358
I20250627 13:55:52.000063 7739 maintenance_manager.cc:643] P 83574c1bf8ea45adaff939158f93c096: FlushMRSOp(5184c3f8e465472e9a98c14f8e703415) complete. Timing: real 0.066s user 0.048s sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":363,"dirs.queue_time_us":424,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":1012,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":676,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":785,"threads_started":3,"wal-append.queue_time_us":328}
I20250627 13:55:52.001698 7848 maintenance_manager.cc:419] P 83574c1bf8ea45adaff939158f93c096: Scheduling UndoDeltaBlockGCOp(5184c3f8e465472e9a98c14f8e703415): 4585 bytes on disk
I20250627 13:55:52.002339 7739 maintenance_manager.cc:643] P 83574c1bf8ea45adaff939158f93c096: UndoDeltaBlockGCOp(5184c3f8e465472e9a98c14f8e703415) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20250627 13:55:52.031000 7979 maintenance_manager.cc:419] P a41a89ae8ed743edb2b5cf4490d8c854: Scheduling FlushMRSOp(1dc569a7b12d40058d073f7b81b2601c): perf score=0.033396
I20250627 13:55:52.075362 7870 maintenance_manager.cc:643] P a41a89ae8ed743edb2b5cf4490d8c854: FlushMRSOp(1dc569a7b12d40058d073f7b81b2601c) complete. Timing: real 0.044s user 0.034s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":336,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1349,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":592,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":483,"threads_started":2,"wal-append.queue_time_us":345}
I20250627 13:55:52.076443 7979 maintenance_manager.cc:419] P a41a89ae8ed743edb2b5cf4490d8c854: Scheduling UndoDeltaBlockGCOp(1dc569a7b12d40058d073f7b81b2601c): 4925 bytes on disk
I20250627 13:55:52.077003 7870 maintenance_manager.cc:643] P a41a89ae8ed743edb2b5cf4490d8c854: UndoDeltaBlockGCOp(1dc569a7b12d40058d073f7b81b2601c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20250627 13:55:53.006174 7848 maintenance_manager.cc:419] P 83574c1bf8ea45adaff939158f93c096: Scheduling FlushMRSOp(897206e017ff4305a8cc9092e30f46b6): perf score=0.033336
I20250627 13:55:53.034574 7739 maintenance_manager.cc:643] P 83574c1bf8ea45adaff939158f93c096: FlushMRSOp(897206e017ff4305a8cc9092e30f46b6) complete. Timing: real 0.028s user 0.019s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":319,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":1737,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":588,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":395,"threads_started":2,"wal-append.queue_time_us":261}
I20250627 13:55:53.035490 7848 maintenance_manager.cc:419] P 83574c1bf8ea45adaff939158f93c096: Scheduling UndoDeltaBlockGCOp(897206e017ff4305a8cc9092e30f46b6): 4585 bytes on disk
I20250627 13:55:53.035954 7739 maintenance_manager.cc:643] P 83574c1bf8ea45adaff939158f93c096: UndoDeltaBlockGCOp(897206e017ff4305a8cc9092e30f46b6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20250627 13:55:53.171460 8109 maintenance_manager.cc:419] P 9cf479bdcd0242738d2a77db4ad9f660: Scheduling FlushMRSOp(9196497cb11f4ccbbd62695253d4c8c9): perf score=0.033385
I20250627 13:55:53.210438 8000 maintenance_manager.cc:643] P 9cf479bdcd0242738d2a77db4ad9f660: FlushMRSOp(9196497cb11f4ccbbd62695253d4c8c9) complete. Timing: real 0.039s user 0.022s sys 0.015s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":329,"dirs.run_cpu_time_us":255,"dirs.run_wall_time_us":1432,"drs_written":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4,"lbm_write_time_us":635,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":466,"threads_started":2,"wal-append.queue_time_us":331}
I20250627 13:55:53.211560 8109 maintenance_manager.cc:419] P 9cf479bdcd0242738d2a77db4ad9f660: Scheduling UndoDeltaBlockGCOp(9196497cb11f4ccbbd62695253d4c8c9): 4840 bytes on disk
I20250627 13:55:53.212215 8000 maintenance_manager.cc:643] P 9cf479bdcd0242738d2a77db4ad9f660: UndoDeltaBlockGCOp(9196497cb11f4ccbbd62695253d4c8c9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20250627 13:55:53.213179 8109 maintenance_manager.cc:419] P 9cf479bdcd0242738d2a77db4ad9f660: Scheduling FlushMRSOp(cff879c8e57d42c28440b3bd310f1ba9): perf score=0.033391
I20250627 13:55:53.247673 8000 maintenance_manager.cc:643] P 9cf479bdcd0242738d2a77db4ad9f660: FlushMRSOp(cff879c8e57d42c28440b3bd310f1ba9) complete. Timing: real 0.034s user 0.028s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":60,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":1464,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":694,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":229,"threads_started":1,"wal-append.queue_time_us":327}
I20250627 13:55:53.248674 8109 maintenance_manager.cc:419] P 9cf479bdcd0242738d2a77db4ad9f660: Scheduling UndoDeltaBlockGCOp(cff879c8e57d42c28440b3bd310f1ba9): 4923 bytes on disk
I20250627 13:55:53.249277 8000 maintenance_manager.cc:643] P 9cf479bdcd0242738d2a77db4ad9f660: UndoDeltaBlockGCOp(cff879c8e57d42c28440b3bd310f1ba9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"spinlock_wait_cycles":8960}
I20250627 13:55:54.039492 7848 maintenance_manager.cc:419] P 83574c1bf8ea45adaff939158f93c096: Scheduling FlushMRSOp(c322cfebf12c4fef8d88fbd68e7e8b6f): perf score=0.033341
I20250627 13:55:54.083869 7739 maintenance_manager.cc:643] P 83574c1bf8ea45adaff939158f93c096: FlushMRSOp(c322cfebf12c4fef8d88fbd68e7e8b6f) complete. Timing: real 0.044s user 0.038s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":270,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1514,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":700,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":359,"threads_started":2,"wal-append.queue_time_us":267}
I20250627 13:55:54.084800 7848 maintenance_manager.cc:419] P 83574c1bf8ea45adaff939158f93c096: Scheduling UndoDeltaBlockGCOp(c322cfebf12c4fef8d88fbd68e7e8b6f): 4840 bytes on disk
I20250627 13:55:54.085315 7739 maintenance_manager.cc:643] P 83574c1bf8ea45adaff939158f93c096: UndoDeltaBlockGCOp(c322cfebf12c4fef8d88fbd68e7e8b6f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20250627 13:55:54.163597 8239 maintenance_manager.cc:419] P e4c831df4da841929d13d6c783099b0e: Scheduling FlushMRSOp(8ab02f13e45c48d293e2d2d229d11e27): perf score=0.033379
I20250627 13:55:54.197181 8130 maintenance_manager.cc:643] P e4c831df4da841929d13d6c783099b0e: FlushMRSOp(8ab02f13e45c48d293e2d2d229d11e27) complete. Timing: real 0.033s user 0.026s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":330,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":1659,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":570,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":445,"threads_started":2,"wal-append.queue_time_us":369}
I20250627 13:55:54.198415 8239 maintenance_manager.cc:419] P e4c831df4da841929d13d6c783099b0e: Scheduling UndoDeltaBlockGCOp(8ab02f13e45c48d293e2d2d229d11e27): 4585 bytes on disk
I20250627 13:55:54.198995 8130 maintenance_manager.cc:643] P e4c831df4da841929d13d6c783099b0e: UndoDeltaBlockGCOp(8ab02f13e45c48d293e2d2d229d11e27) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20250627 13:55:54.253176 8109 maintenance_manager.cc:419] P 9cf479bdcd0242738d2a77db4ad9f660: Scheduling FlushMRSOp(118d5145b23f43798e1c727f2a9e9f92): perf score=0.033402
I20250627 13:55:54.286924 8000 maintenance_manager.cc:643] P 9cf479bdcd0242738d2a77db4ad9f660: FlushMRSOp(118d5145b23f43798e1c727f2a9e9f92) complete. Timing: real 0.033s user 0.031s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":305,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1323,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":616,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":418,"threads_started":2,"wal-append.queue_time_us":295}
I20250627 13:55:54.287950 8109 maintenance_manager.cc:419] P 9cf479bdcd0242738d2a77db4ad9f660: Scheduling UndoDeltaBlockGCOp(118d5145b23f43798e1c727f2a9e9f92): 4925 bytes on disk
I20250627 13:55:54.288537 8000 maintenance_manager.cc:643] P 9cf479bdcd0242738d2a77db4ad9f660: UndoDeltaBlockGCOp(118d5145b23f43798e1c727f2a9e9f92) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20250627 13:55:55.041718 8109 maintenance_manager.cc:419] P 9cf479bdcd0242738d2a77db4ad9f660: Scheduling FlushMRSOp(e2bd060a4a9e4765b45b0242ae56e489): perf score=0.033349
I20250627 13:55:55.074343 8000 maintenance_manager.cc:643] P 9cf479bdcd0242738d2a77db4ad9f660: FlushMRSOp(e2bd060a4a9e4765b45b0242ae56e489) complete. Timing: real 0.032s user 0.029s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":335,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":1382,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":721,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":385,"threads_started":2,"wal-append.queue_time_us":250}
I20250627 13:55:55.075425 8109 maintenance_manager.cc:419] P 9cf479bdcd0242738d2a77db4ad9f660: Scheduling UndoDeltaBlockGCOp(e2bd060a4a9e4765b45b0242ae56e489): 4583 bytes on disk
I20250627 13:55:55.075908 8000 maintenance_manager.cc:643] P 9cf479bdcd0242738d2a77db4ad9f660: UndoDeltaBlockGCOp(e2bd060a4a9e4765b45b0242ae56e489) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20250627 13:55:55.086872 7979 maintenance_manager.cc:419] P a41a89ae8ed743edb2b5cf4490d8c854: Scheduling FlushMRSOp(720cd645f0b7414f819fcbff3fd262c3): perf score=0.033366
I20250627 13:55:55.088953 7848 maintenance_manager.cc:419] P 83574c1bf8ea45adaff939158f93c096: Scheduling FlushMRSOp(720cd645f0b7414f819fcbff3fd262c3): perf score=0.033362
I20250627 13:55:55.139958 7870 maintenance_manager.cc:643] P a41a89ae8ed743edb2b5cf4490d8c854: FlushMRSOp(720cd645f0b7414f819fcbff3fd262c3) complete. Timing: real 0.053s user 0.039s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":10102,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":1649,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1050,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":10158,"threads_started":2,"wal-append.queue_time_us":373}
I20250627 13:55:55.141155 7979 maintenance_manager.cc:419] P a41a89ae8ed743edb2b5cf4490d8c854: Scheduling UndoDeltaBlockGCOp(720cd645f0b7414f819fcbff3fd262c3): 4925 bytes on disk
I20250627 13:55:55.141837 7870 maintenance_manager.cc:643] P a41a89ae8ed743edb2b5cf4490d8c854: UndoDeltaBlockGCOp(720cd645f0b7414f819fcbff3fd262c3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":133,"lbm_reads_lt_1ms":4}
I20250627 13:55:55.148061 7739 maintenance_manager.cc:643] P 83574c1bf8ea45adaff939158f93c096: FlushMRSOp(720cd645f0b7414f819fcbff3fd262c3) complete. Timing: real 0.059s user 0.032s sys 0.021s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":8533,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":1320,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":850,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":8610,"threads_started":2,"wal-append.queue_time_us":266}
I20250627 13:55:55.148995 7848 maintenance_manager.cc:419] P 83574c1bf8ea45adaff939158f93c096: Scheduling UndoDeltaBlockGCOp(720cd645f0b7414f819fcbff3fd262c3): 4925 bytes on disk
I20250627 13:55:55.149466 7739 maintenance_manager.cc:643] P 83574c1bf8ea45adaff939158f93c096: UndoDeltaBlockGCOp(720cd645f0b7414f819fcbff3fd262c3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20250627 13:55:56.205209 8239 maintenance_manager.cc:419] P e4c831df4da841929d13d6c783099b0e: Scheduling FlushMRSOp(c2468f4746224f0ab6f4ba4b3a628e35): perf score=0.033340
I20250627 13:55:56.263988 8130 maintenance_manager.cc:643] P e4c831df4da841929d13d6c783099b0e: FlushMRSOp(c2468f4746224f0ab6f4ba4b3a628e35) complete. Timing: real 0.058s user 0.043s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":377,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":1620,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":692,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":449,"threads_started":2,"wal-append.queue_time_us":339}
I20250627 13:55:56.265053 8239 maintenance_manager.cc:419] P e4c831df4da841929d13d6c783099b0e: Scheduling UndoDeltaBlockGCOp(c2468f4746224f0ab6f4ba4b3a628e35): 4585 bytes on disk
I20250627 13:55:56.265667 8130 maintenance_manager.cc:643] P e4c831df4da841929d13d6c783099b0e: UndoDeltaBlockGCOp(c2468f4746224f0ab6f4ba4b3a628e35) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20250627 13:55:56.330821 8109 maintenance_manager.cc:419] P 9cf479bdcd0242738d2a77db4ad9f660: Scheduling FlushMRSOp(570b134177314b31851951c411a9f4bf): perf score=0.033369
I20250627 13:55:56.370213 8000 maintenance_manager.cc:643] P 9cf479bdcd0242738d2a77db4ad9f660: FlushMRSOp(570b134177314b31851951c411a9f4bf) complete. Timing: real 0.039s user 0.028s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":337,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":1327,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":637,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":2559,"threads_started":2,"wal-append.queue_time_us":2420}
I20250627 13:55:56.371212 8109 maintenance_manager.cc:419] P 9cf479bdcd0242738d2a77db4ad9f660: Scheduling UndoDeltaBlockGCOp(570b134177314b31851951c411a9f4bf): 4840 bytes on disk
I20250627 13:55:56.371820 8000 maintenance_manager.cc:643] P 9cf479bdcd0242738d2a77db4ad9f660: UndoDeltaBlockGCOp(570b134177314b31851951c411a9f4bf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20250627 13:55:56.396247 7979 maintenance_manager.cc:419] P a41a89ae8ed743edb2b5cf4490d8c854: Scheduling FlushMRSOp(91664403eed74badbeba83d69f597e1f): perf score=0.033392
I20250627 13:55:56.403487 7848 maintenance_manager.cc:419] P 83574c1bf8ea45adaff939158f93c096: Scheduling FlushMRSOp(c2468f4746224f0ab6f4ba4b3a628e35): perf score=0.033389
I20250627 13:55:56.433949 7739 maintenance_manager.cc:643] P 83574c1bf8ea45adaff939158f93c096: FlushMRSOp(c2468f4746224f0ab6f4ba4b3a628e35) complete. Timing: real 0.030s user 0.019s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":350,"dirs.run_cpu_time_us":158,"dirs.run_wall_time_us":1304,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":521,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":502,"threads_started":2,"wal-append.queue_time_us":337}
I20250627 13:55:56.434598 7870 maintenance_manager.cc:643] P a41a89ae8ed743edb2b5cf4490d8c854: FlushMRSOp(91664403eed74badbeba83d69f597e1f) complete. Timing: real 0.038s user 0.029s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":1176,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1565,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":744,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1288,"threads_started":2,"wal-append.queue_time_us":338}
I20250627 13:55:56.435043 7848 maintenance_manager.cc:419] P 83574c1bf8ea45adaff939158f93c096: Scheduling UndoDeltaBlockGCOp(c2468f4746224f0ab6f4ba4b3a628e35): 4585 bytes on disk
I20250627 13:55:56.435662 7979 maintenance_manager.cc:419] P a41a89ae8ed743edb2b5cf4490d8c854: Scheduling UndoDeltaBlockGCOp(91664403eed74badbeba83d69f597e1f): 4925 bytes on disk
I20250627 13:55:56.436239 7739 maintenance_manager.cc:643] P 83574c1bf8ea45adaff939158f93c096: UndoDeltaBlockGCOp(c2468f4746224f0ab6f4ba4b3a628e35) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20250627 13:55:56.436239 7870 maintenance_manager.cc:643] P a41a89ae8ed743edb2b5cf4490d8c854: UndoDeltaBlockGCOp(91664403eed74badbeba83d69f597e1f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20250627 13:55:58.273049 8239 maintenance_manager.cc:419] P e4c831df4da841929d13d6c783099b0e: Scheduling FlushMRSOp(00feb28e9047435dbe36a595b1e23fdd): perf score=0.033339
I20250627 13:55:58.307265 8130 maintenance_manager.cc:643] P e4c831df4da841929d13d6c783099b0e: FlushMRSOp(00feb28e9047435dbe36a595b1e23fdd) complete. Timing: real 0.034s user 0.025s sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":285,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1445,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":547,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":427,"threads_started":2,"wal-append.queue_time_us":318}
I20250627 13:55:58.308401 8239 maintenance_manager.cc:419] P e4c831df4da841929d13d6c783099b0e: Scheduling UndoDeltaBlockGCOp(00feb28e9047435dbe36a595b1e23fdd): 4840 bytes on disk
I20250627 13:55:58.308939 8130 maintenance_manager.cc:643] P e4c831df4da841929d13d6c783099b0e: UndoDeltaBlockGCOp(00feb28e9047435dbe36a595b1e23fdd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20250627 13:55:58.378552 8109 maintenance_manager.cc:419] P 9cf479bdcd0242738d2a77db4ad9f660: Scheduling FlushMRSOp(00feb28e9047435dbe36a595b1e23fdd): perf score=0.033371
I20250627 13:55:58.407940 8000 maintenance_manager.cc:643] P 9cf479bdcd0242738d2a77db4ad9f660: FlushMRSOp(00feb28e9047435dbe36a595b1e23fdd) complete. Timing: real 0.029s user 0.022s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":288,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":1284,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":583,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":384,"threads_started":2,"wal-append.queue_time_us":272}
I20250627 13:55:58.408874 8109 maintenance_manager.cc:419] P 9cf479bdcd0242738d2a77db4ad9f660: Scheduling UndoDeltaBlockGCOp(00feb28e9047435dbe36a595b1e23fdd): 4840 bytes on disk
I20250627 13:55:58.409323 8000 maintenance_manager.cc:643] P 9cf479bdcd0242738d2a77db4ad9f660: UndoDeltaBlockGCOp(00feb28e9047435dbe36a595b1e23fdd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20250627 13:55:58.443048 7848 maintenance_manager.cc:419] P 83574c1bf8ea45adaff939158f93c096: Scheduling FlushMRSOp(ff83fdbab2b9417eac50495c54f0da54): perf score=0.033388
I20250627 13:55:58.443106 7979 maintenance_manager.cc:419] P a41a89ae8ed743edb2b5cf4490d8c854: Scheduling FlushMRSOp(72db4bf98ab04b7189640f658d8fdcd7): perf score=0.033401
I20250627 13:55:58.488637 7739 maintenance_manager.cc:643] P 83574c1bf8ea45adaff939158f93c096: FlushMRSOp(ff83fdbab2b9417eac50495c54f0da54) complete. Timing: real 0.045s user 0.034s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":345,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1314,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":836,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":461,"threads_started":2,"wal-append.queue_time_us":311}
I20250627 13:55:58.489930 7848 maintenance_manager.cc:419] P 83574c1bf8ea45adaff939158f93c096: Scheduling UndoDeltaBlockGCOp(ff83fdbab2b9417eac50495c54f0da54): 4583 bytes on disk
I20250627 13:55:58.490429 7739 maintenance_manager.cc:643] P 83574c1bf8ea45adaff939158f93c096: UndoDeltaBlockGCOp(ff83fdbab2b9417eac50495c54f0da54) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20250627 13:55:58.508908 7870 maintenance_manager.cc:643] P a41a89ae8ed743edb2b5cf4490d8c854: FlushMRSOp(72db4bf98ab04b7189640f658d8fdcd7) complete. Timing: real 0.066s user 0.045s sys 0.015s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":279,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1268,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":850,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":351,"threads_started":2,"wal-append.queue_time_us":251}
I20250627 13:55:58.510082 7979 maintenance_manager.cc:419] P a41a89ae8ed743edb2b5cf4490d8c854: Scheduling UndoDeltaBlockGCOp(72db4bf98ab04b7189640f658d8fdcd7): 4925 bytes on disk
I20250627 13:55:58.510558 7870 maintenance_manager.cc:643] P a41a89ae8ed743edb2b5cf4490d8c854: UndoDeltaBlockGCOp(72db4bf98ab04b7189640f658d8fdcd7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20250627 13:55:59.664000 8109 maintenance_manager.cc:419] P 9cf479bdcd0242738d2a77db4ad9f660: Scheduling FlushMRSOp(96fe0309adc54fb6a426bae02f26f898): perf score=0.033336
I20250627 13:55:59.692430 8000 maintenance_manager.cc:643] P 9cf479bdcd0242738d2a77db4ad9f660: FlushMRSOp(96fe0309adc54fb6a426bae02f26f898) complete. Timing: real 0.028s user 0.017s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":348,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":1327,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":560,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":394,"threads_started":2,"wal-append.queue_time_us":241}
I20250627 13:55:59.693353 8109 maintenance_manager.cc:419] P 9cf479bdcd0242738d2a77db4ad9f660: Scheduling UndoDeltaBlockGCOp(96fe0309adc54fb6a426bae02f26f898): 4585 bytes on disk
I20250627 13:55:59.693799 8000 maintenance_manager.cc:643] P 9cf479bdcd0242738d2a77db4ad9f660: UndoDeltaBlockGCOp(96fe0309adc54fb6a426bae02f26f898) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20250627 13:55:59.744894 7848 maintenance_manager.cc:419] P 83574c1bf8ea45adaff939158f93c096: Scheduling FlushMRSOp(eed3236c4fe8455cb9527b9f4b754e81): perf score=0.033360
I20250627 13:55:59.765311 7979 maintenance_manager.cc:419] P a41a89ae8ed743edb2b5cf4490d8c854: Scheduling FlushMRSOp(83cfc975906745a09c5e1dcfac812ae2): perf score=0.033368
I20250627 13:55:59.775347 7739 maintenance_manager.cc:643] P 83574c1bf8ea45adaff939158f93c096: FlushMRSOp(eed3236c4fe8455cb9527b9f4b754e81) complete. Timing: real 0.030s user 0.025s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":335,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1464,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":657,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":457,"threads_started":2,"wal-append.queue_time_us":291}
I20250627 13:55:59.776449 7848 maintenance_manager.cc:419] P 83574c1bf8ea45adaff939158f93c096: Scheduling UndoDeltaBlockGCOp(eed3236c4fe8455cb9527b9f4b754e81): 4840 bytes on disk
I20250627 13:55:59.776947 7739 maintenance_manager.cc:643] P 83574c1bf8ea45adaff939158f93c096: UndoDeltaBlockGCOp(eed3236c4fe8455cb9527b9f4b754e81) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
I20250627 13:55:59.797434 7870 maintenance_manager.cc:643] P a41a89ae8ed743edb2b5cf4490d8c854: FlushMRSOp(83cfc975906745a09c5e1dcfac812ae2) complete. Timing: real 0.032s user 0.029s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":317,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":1256,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":561,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":382,"threads_started":2,"wal-append.queue_time_us":270}
I20250627 13:55:59.798362 7979 maintenance_manager.cc:419] P a41a89ae8ed743edb2b5cf4490d8c854: Scheduling UndoDeltaBlockGCOp(83cfc975906745a09c5e1dcfac812ae2): 4925 bytes on disk
I20250627 13:55:59.798837 7870 maintenance_manager.cc:643] P a41a89ae8ed743edb2b5cf4490d8c854: UndoDeltaBlockGCOp(83cfc975906745a09c5e1dcfac812ae2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20250627 13:55:59.814049 8239 maintenance_manager.cc:419] P e4c831df4da841929d13d6c783099b0e: Scheduling FlushMRSOp(96fe0309adc54fb6a426bae02f26f898): perf score=0.033386
I20250627 13:55:59.842404 8130 maintenance_manager.cc:643] P e4c831df4da841929d13d6c783099b0e: FlushMRSOp(96fe0309adc54fb6a426bae02f26f898) complete. Timing: real 0.028s user 0.027s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":316,"dirs.run_cpu_time_us":135,"dirs.run_wall_time_us":1245,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":496,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":391,"threads_started":2,"wal-append.queue_time_us":256}
I20250627 13:55:59.843330 8239 maintenance_manager.cc:419] P e4c831df4da841929d13d6c783099b0e: Scheduling UndoDeltaBlockGCOp(96fe0309adc54fb6a426bae02f26f898): 4585 bytes on disk
I20250627 13:55:59.843744 8130 maintenance_manager.cc:643] P e4c831df4da841929d13d6c783099b0e: UndoDeltaBlockGCOp(96fe0309adc54fb6a426bae02f26f898) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20250627 13:56:09.177497 9757 maintenance_manager.cc:419] P 2a1282ebe92a4c9a92eed3b13f69012a: Scheduling FlushMRSOp(e3053b529a264e8697a1df9e17eeea81): perf score=0.033351
I20250627 13:56:09.251363 9648 maintenance_manager.cc:643] P 2a1282ebe92a4c9a92eed3b13f69012a: FlushMRSOp(e3053b529a264e8697a1df9e17eeea81) complete. Timing: real 0.073s user 0.054s sys 0.013s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":296,"dirs.queue_time_us":2862,"dirs.run_cpu_time_us":264,"dirs.run_wall_time_us":15076,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":830,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":3150,"threads_started":3,"wal-append.queue_time_us":331}
I20250627 13:56:09.252840 9757 maintenance_manager.cc:419] P 2a1282ebe92a4c9a92eed3b13f69012a: Scheduling UndoDeltaBlockGCOp(e3053b529a264e8697a1df9e17eeea81): 4924 bytes on disk
I20250627 13:56:09.253543 9648 maintenance_manager.cc:643] P 2a1282ebe92a4c9a92eed3b13f69012a: UndoDeltaBlockGCOp(e3053b529a264e8697a1df9e17eeea81) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20250627 13:56:09.560751 10150 maintenance_manager.cc:419] P dede81a8c0164427a61532d01b6fc157: Scheduling FlushMRSOp(d2eb537d49804e1b8ec5a217b1564988): perf score=0.033350
I20250627 13:56:09.683925 10041 maintenance_manager.cc:643] P dede81a8c0164427a61532d01b6fc157: FlushMRSOp(d2eb537d49804e1b8ec5a217b1564988) complete. Timing: real 0.122s user 0.090s sys 0.017s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":429,"dirs.queue_time_us":7824,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":5081,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":775,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":25847,"threads_started":3,"wal-append.queue_time_us":17638}
I20250627 13:56:09.686707 10150 maintenance_manager.cc:419] P dede81a8c0164427a61532d01b6fc157: Scheduling UndoDeltaBlockGCOp(d2eb537d49804e1b8ec5a217b1564988): 4585 bytes on disk
I20250627 13:56:09.687441 10041 maintenance_manager.cc:643] P dede81a8c0164427a61532d01b6fc157: UndoDeltaBlockGCOp(d2eb537d49804e1b8ec5a217b1564988) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20250627 13:56:09.688155 10150 maintenance_manager.cc:419] P dede81a8c0164427a61532d01b6fc157: Scheduling FlushMRSOp(91664403eed74badbeba83d69f597e1f): perf score=0.033336
I20250627 13:56:09.726912 10041 maintenance_manager.cc:643] P dede81a8c0164427a61532d01b6fc157: FlushMRSOp(91664403eed74badbeba83d69f597e1f) complete. Timing: real 0.039s user 0.025s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":67,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1827,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":604,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":293,"threads_started":1,"wal-append.queue_time_us":397}
I20250627 13:56:09.728050 10150 maintenance_manager.cc:419] P dede81a8c0164427a61532d01b6fc157: Scheduling UndoDeltaBlockGCOp(91664403eed74badbeba83d69f597e1f): 4925 bytes on disk
I20250627 13:56:09.728529 10041 maintenance_manager.cc:643] P dede81a8c0164427a61532d01b6fc157: UndoDeltaBlockGCOp(91664403eed74badbeba83d69f597e1f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20250627 13:56:10.006946 9757 maintenance_manager.cc:419] P 2a1282ebe92a4c9a92eed3b13f69012a: Scheduling FlushMRSOp(87726824b05a4b968386f57d3a7a4aad): perf score=0.033390
I20250627 13:56:10.016712 10020 maintenance_manager.cc:419] P 6686c6c365bf4f4582058890161676fe: Scheduling FlushMRSOp(720cd645f0b7414f819fcbff3fd262c3): perf score=0.033337
I20250627 13:56:10.083693 9648 maintenance_manager.cc:643] P 2a1282ebe92a4c9a92eed3b13f69012a: FlushMRSOp(87726824b05a4b968386f57d3a7a4aad) complete. Timing: real 0.076s user 0.041s sys 0.023s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":12093,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":4110,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":744,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":20044,"threads_started":2,"wal-append.queue_time_us":6312}
I20250627 13:56:10.085161 9757 maintenance_manager.cc:419] P 2a1282ebe92a4c9a92eed3b13f69012a: Scheduling UndoDeltaBlockGCOp(87726824b05a4b968386f57d3a7a4aad): 4840 bytes on disk
I20250627 13:56:10.085917 9648 maintenance_manager.cc:643] P 2a1282ebe92a4c9a92eed3b13f69012a: UndoDeltaBlockGCOp(87726824b05a4b968386f57d3a7a4aad) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
I20250627 13:56:10.087134 9757 maintenance_manager.cc:419] P 2a1282ebe92a4c9a92eed3b13f69012a: Scheduling FlushMRSOp(c322cfebf12c4fef8d88fbd68e7e8b6f): perf score=0.033382
I20250627 13:56:10.121621 9911 maintenance_manager.cc:643] P 6686c6c365bf4f4582058890161676fe: FlushMRSOp(720cd645f0b7414f819fcbff3fd262c3) complete. Timing: real 0.104s user 0.059s sys 0.023s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":429,"dirs.queue_time_us":9694,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":2316,"drs_written":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4,"lbm_write_time_us":899,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":13502,"threads_started":3,"wal-append.queue_time_us":3551}
I20250627 13:56:10.124563 10020 maintenance_manager.cc:419] P 6686c6c365bf4f4582058890161676fe: Scheduling FlushMRSOp(ff83fdbab2b9417eac50495c54f0da54): perf score=0.033365
I20250627 13:56:10.164279 9889 maintenance_manager.cc:419] P 956ff058e6d64579a1963aabdb6d466b: Scheduling FlushMRSOp(570b134177314b31851951c411a9f4bf): perf score=0.033361
I20250627 13:56:10.164786 9648 maintenance_manager.cc:643] P 2a1282ebe92a4c9a92eed3b13f69012a: FlushMRSOp(c322cfebf12c4fef8d88fbd68e7e8b6f) complete. Timing: real 0.077s user 0.036s sys 0.025s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":151,"dirs.run_wall_time_us":1243,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":746,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":18557,"threads_started":1,"wal-append.queue_time_us":18648}
I20250627 13:56:10.167829 9757 maintenance_manager.cc:419] P 2a1282ebe92a4c9a92eed3b13f69012a: Scheduling UndoDeltaBlockGCOp(c322cfebf12c4fef8d88fbd68e7e8b6f): 4840 bytes on disk
I20250627 13:56:10.168484 9648 maintenance_manager.cc:643] P 2a1282ebe92a4c9a92eed3b13f69012a: UndoDeltaBlockGCOp(c322cfebf12c4fef8d88fbd68e7e8b6f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20250627 13:56:10.230619 10150 maintenance_manager.cc:419] P dede81a8c0164427a61532d01b6fc157: Scheduling FlushMRSOp(14dba72d45694defb83f41c1572722ca): perf score=0.033378
I20250627 13:56:10.255198 9911 maintenance_manager.cc:643] P 6686c6c365bf4f4582058890161676fe: FlushMRSOp(ff83fdbab2b9417eac50495c54f0da54) complete. Timing: real 0.130s user 0.056s sys 0.017s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1341,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":868,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":388480,"thread_start_us":9143,"threads_started":1,"wal-append.queue_time_us":9194}
I20250627 13:56:10.256667 10020 maintenance_manager.cc:419] P 6686c6c365bf4f4582058890161676fe: Scheduling UndoDeltaBlockGCOp(720cd645f0b7414f819fcbff3fd262c3): 4925 bytes on disk
I20250627 13:56:10.258251 9911 maintenance_manager.cc:643] P 6686c6c365bf4f4582058890161676fe: UndoDeltaBlockGCOp(720cd645f0b7414f819fcbff3fd262c3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20250627 13:56:10.259363 10020 maintenance_manager.cc:419] P 6686c6c365bf4f4582058890161676fe: Scheduling UndoDeltaBlockGCOp(ff83fdbab2b9417eac50495c54f0da54): 4583 bytes on disk
I20250627 13:56:10.259903 9911 maintenance_manager.cc:643] P 6686c6c365bf4f4582058890161676fe: UndoDeltaBlockGCOp(ff83fdbab2b9417eac50495c54f0da54) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20250627 13:56:10.306052 9779 maintenance_manager.cc:643] P 956ff058e6d64579a1963aabdb6d466b: FlushMRSOp(570b134177314b31851951c411a9f4bf) complete. Timing: real 0.141s user 0.069s sys 0.040s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":1671,"dirs.queue_time_us":26509,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":1545,"drs_written":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4,"lbm_write_time_us":905,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":51328,"threads_started":3,"wal-append.queue_time_us":23505}
I20250627 13:56:10.310772 9889 maintenance_manager.cc:419] P 956ff058e6d64579a1963aabdb6d466b: Scheduling UndoDeltaBlockGCOp(570b134177314b31851951c411a9f4bf): 4840 bytes on disk
I20250627 13:56:10.312053 9779 maintenance_manager.cc:643] P 956ff058e6d64579a1963aabdb6d466b: UndoDeltaBlockGCOp(570b134177314b31851951c411a9f4bf) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
I20250627 13:56:10.334978 10041 maintenance_manager.cc:643] P dede81a8c0164427a61532d01b6fc157: FlushMRSOp(14dba72d45694defb83f41c1572722ca) complete. Timing: real 0.104s user 0.058s sys 0.017s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":19005,"dirs.run_cpu_time_us":264,"dirs.run_wall_time_us":4704,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":873,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":29127,"threads_started":2,"wal-append.queue_time_us":10081}
I20250627 13:56:10.336148 10150 maintenance_manager.cc:419] P dede81a8c0164427a61532d01b6fc157: Scheduling UndoDeltaBlockGCOp(14dba72d45694defb83f41c1572722ca): 4840 bytes on disk
I20250627 13:56:10.338320 10041 maintenance_manager.cc:643] P dede81a8c0164427a61532d01b6fc157: UndoDeltaBlockGCOp(14dba72d45694defb83f41c1572722ca) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20250627 13:56:10.420753 9757 maintenance_manager.cc:419] P 2a1282ebe92a4c9a92eed3b13f69012a: Scheduling FlushMRSOp(b265cfda4bfe48908659208ec8eeab2d): perf score=0.033383
I20250627 13:56:10.465590 9648 maintenance_manager.cc:643] P 2a1282ebe92a4c9a92eed3b13f69012a: FlushMRSOp(b265cfda4bfe48908659208ec8eeab2d) complete. Timing: real 0.045s user 0.028s sys 0.011s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":71,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":1283,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2276,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":302,"threads_started":1,"wal-append.queue_time_us":408}
I20250627 13:56:10.466853 9757 maintenance_manager.cc:419] P 2a1282ebe92a4c9a92eed3b13f69012a: Scheduling UndoDeltaBlockGCOp(b265cfda4bfe48908659208ec8eeab2d): 4840 bytes on disk
I20250627 13:56:10.467386 9648 maintenance_manager.cc:643] P 2a1282ebe92a4c9a92eed3b13f69012a: UndoDeltaBlockGCOp(b265cfda4bfe48908659208ec8eeab2d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20250627 13:56:10.519021 10020 maintenance_manager.cc:419] P 6686c6c365bf4f4582058890161676fe: Scheduling FlushMRSOp(40431e3bd7244904a4e65f59477bd21d): perf score=0.033342
I20250627 13:56:10.590507 10150 maintenance_manager.cc:419] P dede81a8c0164427a61532d01b6fc157: Scheduling FlushMRSOp(6f5e370b77cc42fdb0b886ea19af06c3): perf score=0.033345
I20250627 13:56:10.597071 9911 maintenance_manager.cc:643] P 6686c6c365bf4f4582058890161676fe: FlushMRSOp(40431e3bd7244904a4e65f59477bd21d) complete. Timing: real 0.078s user 0.045s sys 0.027s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":78,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1567,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":936,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":6804,"threads_started":1,"wal-append.queue_time_us":6694}
I20250627 13:56:10.598734 10020 maintenance_manager.cc:419] P 6686c6c365bf4f4582058890161676fe: Scheduling UndoDeltaBlockGCOp(40431e3bd7244904a4e65f59477bd21d): 4925 bytes on disk
I20250627 13:56:10.609933 9911 maintenance_manager.cc:643] P 6686c6c365bf4f4582058890161676fe: UndoDeltaBlockGCOp(40431e3bd7244904a4e65f59477bd21d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
I20250627 13:56:10.690268 10041 maintenance_manager.cc:643] P dede81a8c0164427a61532d01b6fc157: FlushMRSOp(6f5e370b77cc42fdb0b886ea19af06c3) complete. Timing: real 0.099s user 0.043s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1267,"drs_written":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1065,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":5214,"threads_started":1,"wal-append.queue_time_us":5331}
I20250627 13:56:10.691655 10150 maintenance_manager.cc:419] P dede81a8c0164427a61532d01b6fc157: Scheduling UndoDeltaBlockGCOp(6f5e370b77cc42fdb0b886ea19af06c3): 4840 bytes on disk
I20250627 13:56:10.692306 10041 maintenance_manager.cc:643] P dede81a8c0164427a61532d01b6fc157: UndoDeltaBlockGCOp(6f5e370b77cc42fdb0b886ea19af06c3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":129,"lbm_reads_lt_1ms":4}
I20250627 13:56:10.719365 9757 maintenance_manager.cc:419] P 2a1282ebe92a4c9a92eed3b13f69012a: Scheduling FlushMRSOp(eed3236c4fe8455cb9527b9f4b754e81): perf score=0.033334
I20250627 13:56:10.760852 9648 maintenance_manager.cc:643] P 2a1282ebe92a4c9a92eed3b13f69012a: FlushMRSOp(eed3236c4fe8455cb9527b9f4b754e81) complete. Timing: real 0.041s user 0.031s sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":82,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":2001,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":683,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":276,"threads_started":1,"wal-append.queue_time_us":390}
I20250627 13:56:10.761886 9757 maintenance_manager.cc:419] P 2a1282ebe92a4c9a92eed3b13f69012a: Scheduling UndoDeltaBlockGCOp(eed3236c4fe8455cb9527b9f4b754e81): 4840 bytes on disk
I20250627 13:56:10.762445 9648 maintenance_manager.cc:643] P 2a1282ebe92a4c9a92eed3b13f69012a: UndoDeltaBlockGCOp(eed3236c4fe8455cb9527b9f4b754e81) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
I20250627 13:56:10.815220 9889 maintenance_manager.cc:419] P 956ff058e6d64579a1963aabdb6d466b: Scheduling FlushMRSOp(da90822ff15e4e5da05114ef38d3970a): perf score=0.033384
I20250627 13:56:10.864676 9779 maintenance_manager.cc:643] P 956ff058e6d64579a1963aabdb6d466b: FlushMRSOp(da90822ff15e4e5da05114ef38d3970a) complete. Timing: real 0.049s user 0.041s sys 0.005s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":24185,"compiler_manager_pool.run_cpu_time_us":38,"compiler_manager_pool.run_wall_time_us":38,"dirs.queue_time_us":286,"dirs.run_cpu_time_us":248,"dirs.run_wall_time_us":1371,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":646,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":387,"threads_started":2,"wal-append.queue_time_us":308}
I20250627 13:56:10.865823 9889 maintenance_manager.cc:419] P 956ff058e6d64579a1963aabdb6d466b: Scheduling UndoDeltaBlockGCOp(da90822ff15e4e5da05114ef38d3970a): 4840 bytes on disk
I20250627 13:56:10.866331 9779 maintenance_manager.cc:643] P 956ff058e6d64579a1963aabdb6d466b: UndoDeltaBlockGCOp(da90822ff15e4e5da05114ef38d3970a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20250627 13:56:11.615002 10020 maintenance_manager.cc:419] P 6686c6c365bf4f4582058890161676fe: Scheduling FlushMRSOp(96fe0309adc54fb6a426bae02f26f898): perf score=0.033362
I20250627 13:56:11.644629 9911 maintenance_manager.cc:643] P 6686c6c365bf4f4582058890161676fe: FlushMRSOp(96fe0309adc54fb6a426bae02f26f898) complete. Timing: real 0.029s user 0.024s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":435,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":1455,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":577,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":572,"threads_started":2,"wal-append.queue_time_us":342}
I20250627 13:56:11.645618 10020 maintenance_manager.cc:419] P 6686c6c365bf4f4582058890161676fe: Scheduling UndoDeltaBlockGCOp(96fe0309adc54fb6a426bae02f26f898): 4585 bytes on disk
I20250627 13:56:11.646144 9911 maintenance_manager.cc:643] P 6686c6c365bf4f4582058890161676fe: UndoDeltaBlockGCOp(96fe0309adc54fb6a426bae02f26f898) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20250627 13:56:11.897569 10020 maintenance_manager.cc:419] P 6686c6c365bf4f4582058890161676fe: Scheduling FlushMRSOp(5184c3f8e465472e9a98c14f8e703415): perf score=0.033344
I20250627 13:56:11.934111 9911 maintenance_manager.cc:643] P 6686c6c365bf4f4582058890161676fe: FlushMRSOp(5184c3f8e465472e9a98c14f8e703415) complete. Timing: real 0.036s user 0.031s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":1427,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":613,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":167,"threads_started":1,"wal-append.queue_time_us":265}
I20250627 13:56:11.935422 10020 maintenance_manager.cc:419] P 6686c6c365bf4f4582058890161676fe: Scheduling UndoDeltaBlockGCOp(5184c3f8e465472e9a98c14f8e703415): 4585 bytes on disk
I20250627 13:56:11.935961 9911 maintenance_manager.cc:643] P 6686c6c365bf4f4582058890161676fe: UndoDeltaBlockGCOp(5184c3f8e465472e9a98c14f8e703415) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20250627 13:56:13.270475 9757 maintenance_manager.cc:419] P 2a1282ebe92a4c9a92eed3b13f69012a: Scheduling FlushMRSOp(72db4bf98ab04b7189640f658d8fdcd7): perf score=0.033347
I20250627 13:56:13.304101 9648 maintenance_manager.cc:643] P 2a1282ebe92a4c9a92eed3b13f69012a: FlushMRSOp(72db4bf98ab04b7189640f658d8fdcd7) complete. Timing: real 0.033s user 0.022s sys 0.009s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":417,"dirs.run_cpu_time_us":222,"dirs.run_wall_time_us":1456,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":597,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":489,"threads_started":2,"wal-append.queue_time_us":315}
I20250627 13:56:13.305294 9757 maintenance_manager.cc:419] P 2a1282ebe92a4c9a92eed3b13f69012a: Scheduling UndoDeltaBlockGCOp(72db4bf98ab04b7189640f658d8fdcd7): 4925 bytes on disk
I20250627 13:56:13.305996 9648 maintenance_manager.cc:643] P 2a1282ebe92a4c9a92eed3b13f69012a: UndoDeltaBlockGCOp(72db4bf98ab04b7189640f658d8fdcd7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20250627 13:56:14.695001 10020 maintenance_manager.cc:419] P 6686c6c365bf4f4582058890161676fe: Scheduling FlushMRSOp(8ab02f13e45c48d293e2d2d229d11e27): perf score=0.033366
I20250627 13:56:14.734577 9911 maintenance_manager.cc:643] P 6686c6c365bf4f4582058890161676fe: FlushMRSOp(8ab02f13e45c48d293e2d2d229d11e27) complete. Timing: real 0.039s user 0.026s sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":308,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":1270,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":532,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":416,"threads_started":2,"wal-append.queue_time_us":310}
I20250627 13:56:14.735639 10020 maintenance_manager.cc:419] P 6686c6c365bf4f4582058890161676fe: Scheduling FlushMRSOp(e3053b529a264e8697a1df9e17eeea81): perf score=0.033363
I20250627 13:56:14.766862 9911 maintenance_manager.cc:643] P 6686c6c365bf4f4582058890161676fe: FlushMRSOp(e3053b529a264e8697a1df9e17eeea81) complete. Timing: real 0.031s user 0.018s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1203,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":570,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":205,"threads_started":1,"wal-append.queue_time_us":314}
I20250627 13:56:14.767911 10020 maintenance_manager.cc:419] P 6686c6c365bf4f4582058890161676fe: Scheduling UndoDeltaBlockGCOp(e3053b529a264e8697a1df9e17eeea81): 4924 bytes on disk
I20250627 13:56:14.768378 9911 maintenance_manager.cc:643] P 6686c6c365bf4f4582058890161676fe: UndoDeltaBlockGCOp(e3053b529a264e8697a1df9e17eeea81) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20250627 13:56:14.769085 10020 maintenance_manager.cc:419] P 6686c6c365bf4f4582058890161676fe: Scheduling UndoDeltaBlockGCOp(8ab02f13e45c48d293e2d2d229d11e27): 4585 bytes on disk
I20250627 13:56:14.769457 9911 maintenance_manager.cc:643] P 6686c6c365bf4f4582058890161676fe: UndoDeltaBlockGCOp(8ab02f13e45c48d293e2d2d229d11e27) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20250627 13:56:14.955756 10150 maintenance_manager.cc:419] P dede81a8c0164427a61532d01b6fc157: Scheduling FlushMRSOp(5f176b66fdce4398be3492a5f3b4ea27): perf score=0.033373
I20250627 13:56:14.985215 10041 maintenance_manager.cc:643] P dede81a8c0164427a61532d01b6fc157: FlushMRSOp(5f176b66fdce4398be3492a5f3b4ea27) complete. Timing: real 0.029s user 0.022s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":347,"dirs.run_cpu_time_us":252,"dirs.run_wall_time_us":1419,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":580,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":768,"thread_start_us":415,"threads_started":2,"wal-append.queue_time_us":742}
I20250627 13:56:14.986157 10150 maintenance_manager.cc:419] P dede81a8c0164427a61532d01b6fc157: Scheduling UndoDeltaBlockGCOp(5f176b66fdce4398be3492a5f3b4ea27): 4585 bytes on disk
I20250627 13:56:14.986630 10041 maintenance_manager.cc:643] P dede81a8c0164427a61532d01b6fc157: UndoDeltaBlockGCOp(5f176b66fdce4398be3492a5f3b4ea27) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20250627 13:56:15.271538 10020 maintenance_manager.cc:419] P 6686c6c365bf4f4582058890161676fe: Scheduling FlushMRSOp(91664403eed74badbeba83d69f597e1f): perf score=0.033338
I20250627 13:56:15.307593 9911 maintenance_manager.cc:643] P 6686c6c365bf4f4582058890161676fe: FlushMRSOp(91664403eed74badbeba83d69f597e1f) complete. Timing: real 0.036s user 0.030s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":323,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":1447,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":609,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":653,"threads_started":2,"wal-append.queue_time_us":517}
I20250627 13:56:15.308841 10020 maintenance_manager.cc:419] P 6686c6c365bf4f4582058890161676fe: Scheduling UndoDeltaBlockGCOp(91664403eed74badbeba83d69f597e1f): 4925 bytes on disk
I20250627 13:56:15.309283 9911 maintenance_manager.cc:643] P 6686c6c365bf4f4582058890161676fe: UndoDeltaBlockGCOp(91664403eed74badbeba83d69f597e1f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20250627 13:56:15.381529 9889 maintenance_manager.cc:419] P 956ff058e6d64579a1963aabdb6d466b: Scheduling FlushMRSOp(00feb28e9047435dbe36a595b1e23fdd): perf score=0.033372
I20250627 13:56:15.413204 9779 maintenance_manager.cc:643] P 956ff058e6d64579a1963aabdb6d466b: FlushMRSOp(00feb28e9047435dbe36a595b1e23fdd) complete. Timing: real 0.031s user 0.022s sys 0.006s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":362,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1114,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":616,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":1211,"threads_started":2,"wal-append.queue_time_us":990}
I20250627 13:56:15.414387 9889 maintenance_manager.cc:419] P 956ff058e6d64579a1963aabdb6d466b: Scheduling UndoDeltaBlockGCOp(00feb28e9047435dbe36a595b1e23fdd): 4840 bytes on disk
I20250627 13:56:15.415063 9779 maintenance_manager.cc:643] P 956ff058e6d64579a1963aabdb6d466b: UndoDeltaBlockGCOp(00feb28e9047435dbe36a595b1e23fdd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4}
I20250627 13:56:15.560969 10020 maintenance_manager.cc:419] P 6686c6c365bf4f4582058890161676fe: Scheduling FlushMRSOp(c2468f4746224f0ab6f4ba4b3a628e35): perf score=0.033336
I20250627 13:56:15.563493 9757 maintenance_manager.cc:419] P 2a1282ebe92a4c9a92eed3b13f69012a: Scheduling FlushMRSOp(cff879c8e57d42c28440b3bd310f1ba9): perf score=0.033346
I20250627 13:56:15.595535 9911 maintenance_manager.cc:643] P 6686c6c365bf4f4582058890161676fe: FlushMRSOp(c2468f4746224f0ab6f4ba4b3a628e35) complete. Timing: real 0.034s user 0.029s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":1521,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":685,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1799,"threads_started":1,"wal-append.queue_time_us":1946}
I20250627 13:56:15.596670 10020 maintenance_manager.cc:419] P 6686c6c365bf4f4582058890161676fe: Scheduling UndoDeltaBlockGCOp(c2468f4746224f0ab6f4ba4b3a628e35): 4585 bytes on disk
I20250627 13:56:15.597267 9911 maintenance_manager.cc:643] P 6686c6c365bf4f4582058890161676fe: UndoDeltaBlockGCOp(c2468f4746224f0ab6f4ba4b3a628e35) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
I20250627 13:56:15.618224 9648 maintenance_manager.cc:643] P 2a1282ebe92a4c9a92eed3b13f69012a: FlushMRSOp(cff879c8e57d42c28440b3bd310f1ba9) complete. Timing: real 0.054s user 0.033s sys 0.019s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":421,"dirs.run_cpu_time_us":151,"dirs.run_wall_time_us":1280,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":800,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":489,"threads_started":2,"wal-append.queue_time_us":294}
I20250627 13:56:15.619164 9757 maintenance_manager.cc:419] P 2a1282ebe92a4c9a92eed3b13f69012a: Scheduling UndoDeltaBlockGCOp(cff879c8e57d42c28440b3bd310f1ba9): 4923 bytes on disk
I20250627 13:56:15.619748 9648 maintenance_manager.cc:643] P 2a1282ebe92a4c9a92eed3b13f69012a: UndoDeltaBlockGCOp(cff879c8e57d42c28440b3bd310f1ba9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20250627 13:56:16.372879 9757 maintenance_manager.cc:419] P 2a1282ebe92a4c9a92eed3b13f69012a: Scheduling FlushMRSOp(6f5e370b77cc42fdb0b886ea19af06c3): perf score=0.033372
I20250627 13:56:16.402798 9648 maintenance_manager.cc:643] P 2a1282ebe92a4c9a92eed3b13f69012a: FlushMRSOp(6f5e370b77cc42fdb0b886ea19af06c3) complete. Timing: real 0.030s user 0.028s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":331,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":1277,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":573,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":407,"threads_started":2,"wal-append.queue_time_us":259}
I20250627 13:56:16.403787 9757 maintenance_manager.cc:419] P 2a1282ebe92a4c9a92eed3b13f69012a: Scheduling UndoDeltaBlockGCOp(6f5e370b77cc42fdb0b886ea19af06c3): 4840 bytes on disk
I20250627 13:56:16.404323 9648 maintenance_manager.cc:643] P 2a1282ebe92a4c9a92eed3b13f69012a: UndoDeltaBlockGCOp(6f5e370b77cc42fdb0b886ea19af06c3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20250627 13:56:16.851891 10020 maintenance_manager.cc:419] P 6686c6c365bf4f4582058890161676fe: Scheduling FlushMRSOp(87726824b05a4b968386f57d3a7a4aad): perf score=0.033343
I20250627 13:56:16.881829 9911 maintenance_manager.cc:643] P 6686c6c365bf4f4582058890161676fe: FlushMRSOp(87726824b05a4b968386f57d3a7a4aad) complete. Timing: real 0.030s user 0.026s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":414,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":1401,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":576,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":482,"threads_started":2,"wal-append.queue_time_us":304}
I20250627 13:56:16.882829 10020 maintenance_manager.cc:419] P 6686c6c365bf4f4582058890161676fe: Scheduling UndoDeltaBlockGCOp(87726824b05a4b968386f57d3a7a4aad): 4840 bytes on disk
I20250627 13:56:16.883338 9911 maintenance_manager.cc:643] P 6686c6c365bf4f4582058890161676fe: UndoDeltaBlockGCOp(87726824b05a4b968386f57d3a7a4aad) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20250627 13:56:17.244099 10150 maintenance_manager.cc:419] P dede81a8c0164427a61532d01b6fc157: Scheduling FlushMRSOp(c322cfebf12c4fef8d88fbd68e7e8b6f): perf score=0.033391
I20250627 13:56:17.274537 10041 maintenance_manager.cc:643] P dede81a8c0164427a61532d01b6fc157: FlushMRSOp(c322cfebf12c4fef8d88fbd68e7e8b6f) complete. Timing: real 0.030s user 0.025s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":318,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1390,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":757,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":424,"threads_started":2,"wal-append.queue_time_us":300}
I20250627 13:56:17.275470 10150 maintenance_manager.cc:419] P dede81a8c0164427a61532d01b6fc157: Scheduling UndoDeltaBlockGCOp(c322cfebf12c4fef8d88fbd68e7e8b6f): 4840 bytes on disk
I20250627 13:56:17.275959 10041 maintenance_manager.cc:643] P dede81a8c0164427a61532d01b6fc157: UndoDeltaBlockGCOp(c322cfebf12c4fef8d88fbd68e7e8b6f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20250627 13:56:19.165002 9757 maintenance_manager.cc:419] P 2a1282ebe92a4c9a92eed3b13f69012a: Scheduling FlushMRSOp(570b134177314b31851951c411a9f4bf): perf score=0.033384
I20250627 13:56:19.202371 9648 maintenance_manager.cc:643] P 2a1282ebe92a4c9a92eed3b13f69012a: FlushMRSOp(570b134177314b31851951c411a9f4bf) complete. Timing: real 0.037s user 0.025s sys 0.011s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":267,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":1832,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":571,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":367,"threads_started":2,"wal-append.queue_time_us":276}
I20250627 13:56:19.203291 9757 maintenance_manager.cc:419] P 2a1282ebe92a4c9a92eed3b13f69012a: Scheduling UndoDeltaBlockGCOp(570b134177314b31851951c411a9f4bf): 4840 bytes on disk
I20250627 13:56:19.203722 9648 maintenance_manager.cc:643] P 2a1282ebe92a4c9a92eed3b13f69012a: UndoDeltaBlockGCOp(570b134177314b31851951c411a9f4bf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20250627 13:56:19.282832 10150 maintenance_manager.cc:419] P dede81a8c0164427a61532d01b6fc157: Scheduling FlushMRSOp(72db4bf98ab04b7189640f658d8fdcd7): perf score=0.033395
I20250627 13:56:19.323350 10041 maintenance_manager.cc:643] P dede81a8c0164427a61532d01b6fc157: FlushMRSOp(72db4bf98ab04b7189640f658d8fdcd7) complete. Timing: real 0.040s user 0.033s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":374,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":1415,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":772,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":483,"threads_started":2,"wal-append.queue_time_us":309}
I20250627 13:56:19.324501 10150 maintenance_manager.cc:419] P dede81a8c0164427a61532d01b6fc157: Scheduling UndoDeltaBlockGCOp(72db4bf98ab04b7189640f658d8fdcd7): 4925 bytes on disk
I20250627 13:56:19.325070 10041 maintenance_manager.cc:643] P dede81a8c0164427a61532d01b6fc157: UndoDeltaBlockGCOp(72db4bf98ab04b7189640f658d8fdcd7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20250627 13:56:19.957576 9757 maintenance_manager.cc:419] P 2a1282ebe92a4c9a92eed3b13f69012a: Scheduling FlushMRSOp(e2bd060a4a9e4765b45b0242ae56e489): perf score=0.033335
I20250627 13:56:19.986024 9648 maintenance_manager.cc:643] P 2a1282ebe92a4c9a92eed3b13f69012a: FlushMRSOp(e2bd060a4a9e4765b45b0242ae56e489) complete. Timing: real 0.028s user 0.023s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":321,"dirs.run_cpu_time_us":233,"dirs.run_wall_time_us":1296,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":527,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":414,"threads_started":2,"wal-append.queue_time_us":289}
I20250627 13:56:19.987077 9757 maintenance_manager.cc:419] P 2a1282ebe92a4c9a92eed3b13f69012a: Scheduling UndoDeltaBlockGCOp(e2bd060a4a9e4765b45b0242ae56e489): 4583 bytes on disk
I20250627 13:56:19.987600 9648 maintenance_manager.cc:643] P 2a1282ebe92a4c9a92eed3b13f69012a: UndoDeltaBlockGCOp(e2bd060a4a9e4765b45b0242ae56e489) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20250627 13:56:20.078308 10150 maintenance_manager.cc:419] P dede81a8c0164427a61532d01b6fc157: Scheduling FlushMRSOp(03640a5124d2405faf4a26d0e22e1ffc): perf score=0.033399
I20250627 13:56:20.120216 10041 maintenance_manager.cc:643] P dede81a8c0164427a61532d01b6fc157: FlushMRSOp(03640a5124d2405faf4a26d0e22e1ffc) complete. Timing: real 0.042s user 0.039s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":363,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":1522,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":568,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":448,"threads_started":2,"wal-append.queue_time_us":311}
I20250627 13:56:20.121243 10150 maintenance_manager.cc:419] P dede81a8c0164427a61532d01b6fc157: Scheduling UndoDeltaBlockGCOp(03640a5124d2405faf4a26d0e22e1ffc): 4586 bytes on disk
I20250627 13:56:20.121762 10041 maintenance_manager.cc:643] P dede81a8c0164427a61532d01b6fc157: UndoDeltaBlockGCOp(03640a5124d2405faf4a26d0e22e1ffc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20250627 13:56:20.143915 10020 maintenance_manager.cc:419] P 6686c6c365bf4f4582058890161676fe: Scheduling FlushMRSOp(9196497cb11f4ccbbd62695253d4c8c9): perf score=0.033335
I20250627 13:56:20.180670 9889 maintenance_manager.cc:419] P 956ff058e6d64579a1963aabdb6d466b: Scheduling FlushMRSOp(eed3236c4fe8455cb9527b9f4b754e81): perf score=0.033382
I20250627 13:56:20.180016 9911 maintenance_manager.cc:643] P 6686c6c365bf4f4582058890161676fe: FlushMRSOp(9196497cb11f4ccbbd62695253d4c8c9) complete. Timing: real 0.036s user 0.025s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":315,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":1680,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":889,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":506,"threads_started":2,"wal-append.queue_time_us":413}
I20250627 13:56:20.181847 10020 maintenance_manager.cc:419] P 6686c6c365bf4f4582058890161676fe: Scheduling UndoDeltaBlockGCOp(9196497cb11f4ccbbd62695253d4c8c9): 4840 bytes on disk
I20250627 13:56:20.182318 9911 maintenance_manager.cc:643] P 6686c6c365bf4f4582058890161676fe: UndoDeltaBlockGCOp(9196497cb11f4ccbbd62695253d4c8c9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20250627 13:56:20.211131 9779 maintenance_manager.cc:643] P 956ff058e6d64579a1963aabdb6d466b: FlushMRSOp(eed3236c4fe8455cb9527b9f4b754e81) complete. Timing: real 0.030s user 0.029s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":491,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":1258,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":639,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":428,"threads_started":2,"wal-append.queue_time_us":278}
I20250627 13:56:20.212221 9889 maintenance_manager.cc:419] P 956ff058e6d64579a1963aabdb6d466b: Scheduling UndoDeltaBlockGCOp(eed3236c4fe8455cb9527b9f4b754e81): 4840 bytes on disk
I20250627 13:56:20.212810 9779 maintenance_manager.cc:643] P 956ff058e6d64579a1963aabdb6d466b: UndoDeltaBlockGCOp(eed3236c4fe8455cb9527b9f4b754e81) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20250627 13:56:20.213721 9889 maintenance_manager.cc:419] P 956ff058e6d64579a1963aabdb6d466b: Scheduling FlushMRSOp(5184c3f8e465472e9a98c14f8e703415): perf score=0.033360
I20250627 13:56:20.244100 9779 maintenance_manager.cc:643] P 956ff058e6d64579a1963aabdb6d466b: FlushMRSOp(5184c3f8e465472e9a98c14f8e703415) complete. Timing: real 0.030s user 0.019s sys 0.009s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":73,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1327,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":559,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":207,"threads_started":1,"wal-append.queue_time_us":305}
I20250627 13:56:20.245121 9889 maintenance_manager.cc:419] P 956ff058e6d64579a1963aabdb6d466b: Scheduling UndoDeltaBlockGCOp(5184c3f8e465472e9a98c14f8e703415): 4585 bytes on disk
I20250627 13:56:20.245599 9779 maintenance_manager.cc:643] P 956ff058e6d64579a1963aabdb6d466b: UndoDeltaBlockGCOp(5184c3f8e465472e9a98c14f8e703415) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20250627 13:56:20.373396 10150 maintenance_manager.cc:419] P dede81a8c0164427a61532d01b6fc157: Scheduling FlushMRSOp(ff83fdbab2b9417eac50495c54f0da54): perf score=0.033364
I20250627 13:56:20.403631 10041 maintenance_manager.cc:643] P dede81a8c0164427a61532d01b6fc157: FlushMRSOp(ff83fdbab2b9417eac50495c54f0da54) complete. Timing: real 0.030s user 0.017s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":71,"dirs.run_cpu_time_us":249,"dirs.run_wall_time_us":1390,"drs_written":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4,"lbm_write_time_us":618,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":247,"threads_started":1,"wal-append.queue_time_us":350}
I20250627 13:56:20.404764 10150 maintenance_manager.cc:419] P dede81a8c0164427a61532d01b6fc157: Scheduling UndoDeltaBlockGCOp(ff83fdbab2b9417eac50495c54f0da54): 4583 bytes on disk
I20250627 13:56:20.405370 10041 maintenance_manager.cc:643] P dede81a8c0164427a61532d01b6fc157: UndoDeltaBlockGCOp(ff83fdbab2b9417eac50495c54f0da54) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20250627 13:56:20.999138 9889 maintenance_manager.cc:419] P 956ff058e6d64579a1963aabdb6d466b: Scheduling FlushMRSOp(1dc569a7b12d40058d073f7b81b2601c): perf score=0.033337
I20250627 13:56:21.029872 9779 maintenance_manager.cc:643] P 956ff058e6d64579a1963aabdb6d466b: FlushMRSOp(1dc569a7b12d40058d073f7b81b2601c) complete. Timing: real 0.030s user 0.026s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":287,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":1445,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":607,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":404,"threads_started":2,"wal-append.queue_time_us":305}
I20250627 13:56:21.030829 9889 maintenance_manager.cc:419] P 956ff058e6d64579a1963aabdb6d466b: Scheduling UndoDeltaBlockGCOp(1dc569a7b12d40058d073f7b81b2601c): 4925 bytes on disk
I20250627 13:56:21.031291 9779 maintenance_manager.cc:643] P 956ff058e6d64579a1963aabdb6d466b: UndoDeltaBlockGCOp(1dc569a7b12d40058d073f7b81b2601c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20250627 13:56:21.158344 10150 maintenance_manager.cc:419] P dede81a8c0164427a61532d01b6fc157: Scheduling FlushMRSOp(83cfc975906745a09c5e1dcfac812ae2): perf score=0.033378
I20250627 13:56:21.188614 10041 maintenance_manager.cc:643] P dede81a8c0164427a61532d01b6fc157: FlushMRSOp(83cfc975906745a09c5e1dcfac812ae2) complete. Timing: real 0.030s user 0.022s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":289,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1219,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":630,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":393,"threads_started":2,"wal-append.queue_time_us":280}
I20250627 13:56:21.190106 10150 maintenance_manager.cc:419] P dede81a8c0164427a61532d01b6fc157: Scheduling UndoDeltaBlockGCOp(83cfc975906745a09c5e1dcfac812ae2): 4925 bytes on disk
I20250627 13:56:21.190902 10041 maintenance_manager.cc:643] P dede81a8c0164427a61532d01b6fc157: UndoDeltaBlockGCOp(83cfc975906745a09c5e1dcfac812ae2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20250627 13:56:24.040912 9889 maintenance_manager.cc:419] P 956ff058e6d64579a1963aabdb6d466b: Scheduling FlushMRSOp(118d5145b23f43798e1c727f2a9e9f92): perf score=0.033399
I20250627 13:56:24.070597 9779 maintenance_manager.cc:643] P 956ff058e6d64579a1963aabdb6d466b: FlushMRSOp(118d5145b23f43798e1c727f2a9e9f92) complete. Timing: real 0.029s user 0.025s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":307,"dirs.run_cpu_time_us":297,"dirs.run_wall_time_us":1630,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":560,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":388,"threads_started":2,"wal-append.queue_time_us":267}
I20250627 13:56:24.071493 9889 maintenance_manager.cc:419] P 956ff058e6d64579a1963aabdb6d466b: Scheduling UndoDeltaBlockGCOp(118d5145b23f43798e1c727f2a9e9f92): 4925 bytes on disk
I20250627 13:56:24.071914 9779 maintenance_manager.cc:643] P 956ff058e6d64579a1963aabdb6d466b: UndoDeltaBlockGCOp(118d5145b23f43798e1c727f2a9e9f92) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20250627 13:56:24.119403 7979 maintenance_manager.cc:419] P a41a89ae8ed743edb2b5cf4490d8c854: Scheduling FlushMRSOp(8ab02f13e45c48d293e2d2d229d11e27): perf score=0.033389
I20250627 13:56:24.147020 7870 maintenance_manager.cc:643] P a41a89ae8ed743edb2b5cf4490d8c854: FlushMRSOp(8ab02f13e45c48d293e2d2d229d11e27) complete. Timing: real 0.027s user 0.020s sys 0.006s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":324,"dirs.run_cpu_time_us":138,"dirs.run_wall_time_us":957,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":542,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":390,"threads_started":2,"wal-append.queue_time_us":314}
I20250627 13:56:24.148279 7979 maintenance_manager.cc:419] P a41a89ae8ed743edb2b5cf4490d8c854: Scheduling UndoDeltaBlockGCOp(8ab02f13e45c48d293e2d2d229d11e27): 4585 bytes on disk
I20250627 13:56:24.148885 7870 maintenance_manager.cc:643] P a41a89ae8ed743edb2b5cf4490d8c854: UndoDeltaBlockGCOp(8ab02f13e45c48d293e2d2d229d11e27) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20250627 13:56:24.251114 9757 maintenance_manager.cc:419] P 2a1282ebe92a4c9a92eed3b13f69012a: Scheduling FlushMRSOp(da90822ff15e4e5da05114ef38d3970a): perf score=0.033394
I20250627 13:56:24.279831 9648 maintenance_manager.cc:643] P 2a1282ebe92a4c9a92eed3b13f69012a: FlushMRSOp(da90822ff15e4e5da05114ef38d3970a) complete. Timing: real 0.028s user 0.022s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":294,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1220,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":516,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":354,"threads_started":2,"wal-append.queue_time_us":225}
I20250627 13:56:24.280958 9757 maintenance_manager.cc:419] P 2a1282ebe92a4c9a92eed3b13f69012a: Scheduling UndoDeltaBlockGCOp(da90822ff15e4e5da05114ef38d3970a): 4840 bytes on disk
I20250627 13:56:24.281433 9648 maintenance_manager.cc:643] P 2a1282ebe92a4c9a92eed3b13f69012a: UndoDeltaBlockGCOp(da90822ff15e4e5da05114ef38d3970a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20250627 13:56:24.400620 7979 maintenance_manager.cc:419] P a41a89ae8ed743edb2b5cf4490d8c854: Scheduling FlushMRSOp(118d5145b23f43798e1c727f2a9e9f92): perf score=0.033367
I20250627 13:56:24.435407 7870 maintenance_manager.cc:643] P a41a89ae8ed743edb2b5cf4490d8c854: FlushMRSOp(118d5145b23f43798e1c727f2a9e9f92) complete. Timing: real 0.035s user 0.022s sys 0.010s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":1494,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":692,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":604,"threads_started":1,"wal-append.queue_time_us":695}
I20250627 13:56:24.436383 7979 maintenance_manager.cc:419] P a41a89ae8ed743edb2b5cf4490d8c854: Scheduling UndoDeltaBlockGCOp(118d5145b23f43798e1c727f2a9e9f92): 4925 bytes on disk
I20250627 13:56:24.436815 7870 maintenance_manager.cc:643] P a41a89ae8ed743edb2b5cf4490d8c854: UndoDeltaBlockGCOp(118d5145b23f43798e1c727f2a9e9f92) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20250627 13:56:24.533015 9757 maintenance_manager.cc:419] P 2a1282ebe92a4c9a92eed3b13f69012a: Scheduling FlushMRSOp(40431e3bd7244904a4e65f59477bd21d): perf score=0.033355
I20250627 13:56:24.562805 9648 maintenance_manager.cc:643] P 2a1282ebe92a4c9a92eed3b13f69012a: FlushMRSOp(40431e3bd7244904a4e65f59477bd21d) complete. Timing: real 0.030s user 0.021s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":137,"dirs.run_wall_time_us":1417,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":535,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":202,"threads_started":1,"wal-append.queue_time_us":304}
I20250627 13:56:24.563815 9757 maintenance_manager.cc:419] P 2a1282ebe92a4c9a92eed3b13f69012a: Scheduling UndoDeltaBlockGCOp(40431e3bd7244904a4e65f59477bd21d): 4925 bytes on disk
I20250627 13:56:24.564347 9648 maintenance_manager.cc:643] P 2a1282ebe92a4c9a92eed3b13f69012a: UndoDeltaBlockGCOp(40431e3bd7244904a4e65f59477bd21d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20250627 13:56:27.082052 9889 maintenance_manager.cc:419] P 956ff058e6d64579a1963aabdb6d466b: Scheduling FlushMRSOp(b265cfda4bfe48908659208ec8eeab2d): perf score=0.033357
I20250627 13:56:27.115986 9779 maintenance_manager.cc:643] P 956ff058e6d64579a1963aabdb6d466b: FlushMRSOp(b265cfda4bfe48908659208ec8eeab2d) complete. Timing: real 0.034s user 0.026s sys 0.005s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":304,"dirs.run_cpu_time_us":246,"dirs.run_wall_time_us":1872,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":874,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":378,"threads_started":2,"wal-append.queue_time_us":253}
I20250627 13:56:27.117134 9889 maintenance_manager.cc:419] P 956ff058e6d64579a1963aabdb6d466b: Scheduling UndoDeltaBlockGCOp(b265cfda4bfe48908659208ec8eeab2d): 4840 bytes on disk
I20250627 13:56:27.117622 9779 maintenance_manager.cc:643] P 956ff058e6d64579a1963aabdb6d466b: UndoDeltaBlockGCOp(b265cfda4bfe48908659208ec8eeab2d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20250627 13:56:27.180620 8239 maintenance_manager.cc:419] P e4c831df4da841929d13d6c783099b0e: Scheduling FlushMRSOp(9196497cb11f4ccbbd62695253d4c8c9): perf score=0.033350
I20250627 13:56:27.211979 8130 maintenance_manager.cc:643] P e4c831df4da841929d13d6c783099b0e: FlushMRSOp(9196497cb11f4ccbbd62695253d4c8c9) complete. Timing: real 0.031s user 0.025s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":334,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":1101,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":577,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":449,"threads_started":2,"wal-append.queue_time_us":332}
I20250627 13:56:27.212896 8239 maintenance_manager.cc:419] P e4c831df4da841929d13d6c783099b0e: Scheduling UndoDeltaBlockGCOp(9196497cb11f4ccbbd62695253d4c8c9): 4840 bytes on disk
I20250627 13:56:27.213377 8130 maintenance_manager.cc:643] P e4c831df4da841929d13d6c783099b0e: UndoDeltaBlockGCOp(9196497cb11f4ccbbd62695253d4c8c9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20250627 13:56:27.214147 8239 maintenance_manager.cc:419] P e4c831df4da841929d13d6c783099b0e: Scheduling FlushMRSOp(5f176b66fdce4398be3492a5f3b4ea27): perf score=0.033337
I20250627 13:56:27.270591 8130 maintenance_manager.cc:643] P e4c831df4da841929d13d6c783099b0e: FlushMRSOp(5f176b66fdce4398be3492a5f3b4ea27) complete. Timing: real 0.056s user 0.047s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":80,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":1379,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":598,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":212,"threads_started":1,"wal-append.queue_time_us":336}
I20250627 13:56:27.271499 8239 maintenance_manager.cc:419] P e4c831df4da841929d13d6c783099b0e: Scheduling UndoDeltaBlockGCOp(5f176b66fdce4398be3492a5f3b4ea27): 4585 bytes on disk
I20250627 13:56:27.272064 8130 maintenance_manager.cc:643] P e4c831df4da841929d13d6c783099b0e: UndoDeltaBlockGCOp(5f176b66fdce4398be3492a5f3b4ea27) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20250627 13:56:27.527464 8109 maintenance_manager.cc:419] P 9cf479bdcd0242738d2a77db4ad9f660: Scheduling FlushMRSOp(1dc569a7b12d40058d073f7b81b2601c): perf score=0.033360
I20250627 13:56:27.557981 8000 maintenance_manager.cc:643] P 9cf479bdcd0242738d2a77db4ad9f660: FlushMRSOp(1dc569a7b12d40058d073f7b81b2601c) complete. Timing: real 0.030s user 0.023s sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":330,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":825,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":563,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":407,"threads_started":2,"wal-append.queue_time_us":310}
I20250627 13:56:27.559036 8109 maintenance_manager.cc:419] P 9cf479bdcd0242738d2a77db4ad9f660: Scheduling UndoDeltaBlockGCOp(1dc569a7b12d40058d073f7b81b2601c): 4925 bytes on disk
I20250627 13:56:27.559520 8000 maintenance_manager.cc:643] P 9cf479bdcd0242738d2a77db4ad9f660: UndoDeltaBlockGCOp(1dc569a7b12d40058d073f7b81b2601c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20250627 13:56:28.711822 10150 maintenance_manager.cc:419] P dede81a8c0164427a61532d01b6fc157: Scheduling FlushMRSOp(cff879c8e57d42c28440b3bd310f1ba9): perf score=0.033342
I20250627 13:56:28.760967 10041 maintenance_manager.cc:643] P dede81a8c0164427a61532d01b6fc157: FlushMRSOp(cff879c8e57d42c28440b3bd310f1ba9) complete. Timing: real 0.049s user 0.039s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":432,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":1423,"drs_written":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4,"lbm_write_time_us":697,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":470,"threads_started":2,"wal-append.queue_time_us":275}
I20250627 13:56:28.762070 10150 maintenance_manager.cc:419] P dede81a8c0164427a61532d01b6fc157: Scheduling UndoDeltaBlockGCOp(cff879c8e57d42c28440b3bd310f1ba9): 4923 bytes on disk
I20250627 13:56:28.762615 10041 maintenance_manager.cc:643] P dede81a8c0164427a61532d01b6fc157: UndoDeltaBlockGCOp(cff879c8e57d42c28440b3bd310f1ba9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20250627 13:56:29.027889 8239 maintenance_manager.cc:419] P e4c831df4da841929d13d6c783099b0e: Scheduling FlushMRSOp(e2bd060a4a9e4765b45b0242ae56e489): perf score=0.033361
I20250627 13:56:29.058975 8130 maintenance_manager.cc:643] P e4c831df4da841929d13d6c783099b0e: FlushMRSOp(e2bd060a4a9e4765b45b0242ae56e489) complete. Timing: real 0.031s user 0.022s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":325,"dirs.run_cpu_time_us":296,"dirs.run_wall_time_us":1632,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":794,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":424,"threads_started":2,"wal-append.queue_time_us":297}
I20250627 13:56:29.060129 8239 maintenance_manager.cc:419] P e4c831df4da841929d13d6c783099b0e: Scheduling UndoDeltaBlockGCOp(e2bd060a4a9e4765b45b0242ae56e489): 4583 bytes on disk
I20250627 13:56:29.060647 8130 maintenance_manager.cc:643] P e4c831df4da841929d13d6c783099b0e: UndoDeltaBlockGCOp(e2bd060a4a9e4765b45b0242ae56e489) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20250627 13:56:29.962687 10020 maintenance_manager.cc:419] P 6686c6c365bf4f4582058890161676fe: Scheduling FlushMRSOp(83cfc975906745a09c5e1dcfac812ae2): perf score=0.033356
I20250627 13:56:30.010777 9911 maintenance_manager.cc:643] P 6686c6c365bf4f4582058890161676fe: FlushMRSOp(83cfc975906745a09c5e1dcfac812ae2) complete. Timing: real 0.029s user 0.028s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":334,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1354,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":594,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":419,"threads_started":2,"wal-append.queue_time_us":301}
I20250627 13:56:30.011693 10020 maintenance_manager.cc:419] P 6686c6c365bf4f4582058890161676fe: Scheduling UndoDeltaBlockGCOp(83cfc975906745a09c5e1dcfac812ae2): 4925 bytes on disk
I20250627 13:56:30.012147 9911 maintenance_manager.cc:643] P 6686c6c365bf4f4582058890161676fe: UndoDeltaBlockGCOp(83cfc975906745a09c5e1dcfac812ae2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20250627 13:56:30.203279 7979 maintenance_manager.cc:419] P a41a89ae8ed743edb2b5cf4490d8c854: Scheduling FlushMRSOp(897206e017ff4305a8cc9092e30f46b6): perf score=0.033363
I20250627 13:56:30.232136 7870 maintenance_manager.cc:643] P a41a89ae8ed743edb2b5cf4490d8c854: FlushMRSOp(897206e017ff4305a8cc9092e30f46b6) complete. Timing: real 0.029s user 0.025s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":447,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1384,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":577,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":526,"threads_started":2,"wal-append.queue_time_us":336}
I20250627 13:56:30.233804 7979 maintenance_manager.cc:419] P a41a89ae8ed743edb2b5cf4490d8c854: Scheduling UndoDeltaBlockGCOp(897206e017ff4305a8cc9092e30f46b6): 4585 bytes on disk
I20250627 13:56:30.234380 7870 maintenance_manager.cc:643] P a41a89ae8ed743edb2b5cf4490d8c854: UndoDeltaBlockGCOp(897206e017ff4305a8cc9092e30f46b6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20250627 13:56:30.620203 7848 maintenance_manager.cc:419] P 83574c1bf8ea45adaff939158f93c096: Scheduling FlushMRSOp(40431e3bd7244904a4e65f59477bd21d): perf score=0.033387
I20250627 13:56:30.628661 9889 maintenance_manager.cc:419] P 956ff058e6d64579a1963aabdb6d466b: Scheduling FlushMRSOp(897206e017ff4305a8cc9092e30f46b6): perf score=0.033358
I20250627 13:56:30.659603 7739 maintenance_manager.cc:643] P 83574c1bf8ea45adaff939158f93c096: FlushMRSOp(40431e3bd7244904a4e65f59477bd21d) complete. Timing: real 0.039s user 0.033s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":361,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":914,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":646,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":426,"threads_started":2,"wal-append.queue_time_us":301}
I20250627 13:56:30.661751 7848 maintenance_manager.cc:419] P 83574c1bf8ea45adaff939158f93c096: Scheduling UndoDeltaBlockGCOp(40431e3bd7244904a4e65f59477bd21d): 4925 bytes on disk
I20250627 13:56:30.662462 7739 maintenance_manager.cc:643] P 83574c1bf8ea45adaff939158f93c096: UndoDeltaBlockGCOp(40431e3bd7244904a4e65f59477bd21d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4}
I20250627 13:56:30.690717 9779 maintenance_manager.cc:643] P 956ff058e6d64579a1963aabdb6d466b: FlushMRSOp(897206e017ff4305a8cc9092e30f46b6) complete. Timing: real 0.062s user 0.040s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":2373,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":1256,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":610,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":2486,"threads_started":2,"wal-append.queue_time_us":316}
I20250627 13:56:30.691756 9889 maintenance_manager.cc:419] P 956ff058e6d64579a1963aabdb6d466b: Scheduling UndoDeltaBlockGCOp(897206e017ff4305a8cc9092e30f46b6): 4585 bytes on disk
I20250627 13:56:30.692219 9779 maintenance_manager.cc:643] P 956ff058e6d64579a1963aabdb6d466b: UndoDeltaBlockGCOp(897206e017ff4305a8cc9092e30f46b6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20250627 13:56:32.324932 8109 maintenance_manager.cc:419] P 9cf479bdcd0242738d2a77db4ad9f660: Scheduling FlushMRSOp(14dba72d45694defb83f41c1572722ca): perf score=0.033377
I20250627 13:56:32.357481 8000 maintenance_manager.cc:643] P 9cf479bdcd0242738d2a77db4ad9f660: FlushMRSOp(14dba72d45694defb83f41c1572722ca) complete. Timing: real 0.032s user 0.026s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":400,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1556,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":635,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":518,"threads_started":2,"wal-append.queue_time_us":335}
I20250627 13:56:32.358518 8109 maintenance_manager.cc:419] P 9cf479bdcd0242738d2a77db4ad9f660: Scheduling UndoDeltaBlockGCOp(14dba72d45694defb83f41c1572722ca): 4840 bytes on disk
I20250627 13:56:32.359071 8000 maintenance_manager.cc:643] P 9cf479bdcd0242738d2a77db4ad9f660: UndoDeltaBlockGCOp(14dba72d45694defb83f41c1572722ca) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20250627 13:56:32.920222 7848 maintenance_manager.cc:419] P 83574c1bf8ea45adaff939158f93c096: Scheduling FlushMRSOp(644985e3086d4614b6598a727cb60c1d): perf score=0.033379
I20250627 13:56:32.949904 7739 maintenance_manager.cc:643] P 83574c1bf8ea45adaff939158f93c096: FlushMRSOp(644985e3086d4614b6598a727cb60c1d) complete. Timing: real 0.029s user 0.021s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":371,"dirs.run_cpu_time_us":251,"dirs.run_wall_time_us":1534,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":592,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":463,"threads_started":2,"wal-append.queue_time_us":311}
I20250627 13:56:32.950896 7848 maintenance_manager.cc:419] P 83574c1bf8ea45adaff939158f93c096: Scheduling UndoDeltaBlockGCOp(644985e3086d4614b6598a727cb60c1d): 4926 bytes on disk
I20250627 13:56:32.951444 7739 maintenance_manager.cc:643] P 83574c1bf8ea45adaff939158f93c096: UndoDeltaBlockGCOp(644985e3086d4614b6598a727cb60c1d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20250627 13:56:33.955183 7848 maintenance_manager.cc:419] P 83574c1bf8ea45adaff939158f93c096: Scheduling FlushMRSOp(d2eb537d49804e1b8ec5a217b1564988): perf score=0.033353
I20250627 13:56:33.984602 7739 maintenance_manager.cc:643] P 83574c1bf8ea45adaff939158f93c096: FlushMRSOp(d2eb537d49804e1b8ec5a217b1564988) complete. Timing: real 0.029s user 0.026s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":292,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":1402,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":574,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":473,"threads_started":2,"wal-append.queue_time_us":413}
I20250627 13:56:33.985713 7848 maintenance_manager.cc:419] P 83574c1bf8ea45adaff939158f93c096: Scheduling UndoDeltaBlockGCOp(d2eb537d49804e1b8ec5a217b1564988): 4585 bytes on disk
I20250627 13:56:33.986203 7739 maintenance_manager.cc:643] P 83574c1bf8ea45adaff939158f93c096: UndoDeltaBlockGCOp(d2eb537d49804e1b8ec5a217b1564988) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20250627 13:56:34.204134 9889 maintenance_manager.cc:419] P 956ff058e6d64579a1963aabdb6d466b: Scheduling FlushMRSOp(e2bd060a4a9e4765b45b0242ae56e489): perf score=0.033399
I20250627 13:56:34.237984 7848 maintenance_manager.cc:419] P 83574c1bf8ea45adaff939158f93c096: Scheduling FlushMRSOp(e2bd060a4a9e4765b45b0242ae56e489): perf score=0.033344
I20250627 13:56:34.239209 9779 maintenance_manager.cc:643] P 956ff058e6d64579a1963aabdb6d466b: FlushMRSOp(e2bd060a4a9e4765b45b0242ae56e489) complete. Timing: real 0.035s user 0.025s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":361,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":1462,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":840,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":472,"threads_started":2,"wal-append.queue_time_us":403}
I20250627 13:56:34.240540 9889 maintenance_manager.cc:419] P 956ff058e6d64579a1963aabdb6d466b: Scheduling UndoDeltaBlockGCOp(e2bd060a4a9e4765b45b0242ae56e489): 4583 bytes on disk
I20250627 13:56:34.241215 9779 maintenance_manager.cc:643] P 956ff058e6d64579a1963aabdb6d466b: UndoDeltaBlockGCOp(e2bd060a4a9e4765b45b0242ae56e489) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20250627 13:56:34.270313 7739 maintenance_manager.cc:643] P 83574c1bf8ea45adaff939158f93c096: FlushMRSOp(e2bd060a4a9e4765b45b0242ae56e489) complete. Timing: real 0.032s user 0.027s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":63,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1238,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":618,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":266,"threads_started":1,"wal-append.queue_time_us":356}
I20250627 13:56:34.272315 7848 maintenance_manager.cc:419] P 83574c1bf8ea45adaff939158f93c096: Scheduling UndoDeltaBlockGCOp(e2bd060a4a9e4765b45b0242ae56e489): 4583 bytes on disk
I20250627 13:56:34.273231 7739 maintenance_manager.cc:643] P 83574c1bf8ea45adaff939158f93c096: UndoDeltaBlockGCOp(e2bd060a4a9e4765b45b0242ae56e489) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":163,"lbm_reads_lt_1ms":4}
I20250627 13:56:36.248948 9889 maintenance_manager.cc:419] P 956ff058e6d64579a1963aabdb6d466b: Scheduling FlushMRSOp(14dba72d45694defb83f41c1572722ca): perf score=0.033400
I20250627 13:56:36.280539 9779 maintenance_manager.cc:643] P 956ff058e6d64579a1963aabdb6d466b: FlushMRSOp(14dba72d45694defb83f41c1572722ca) complete. Timing: real 0.031s user 0.023s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":333,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":1709,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":608,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":552,"threads_started":2,"wal-append.queue_time_us":435}
I20250627 13:56:36.281559 9889 maintenance_manager.cc:419] P 956ff058e6d64579a1963aabdb6d466b: Scheduling UndoDeltaBlockGCOp(14dba72d45694defb83f41c1572722ca): 4840 bytes on disk
I20250627 13:56:36.282061 9779 maintenance_manager.cc:643] P 956ff058e6d64579a1963aabdb6d466b: UndoDeltaBlockGCOp(14dba72d45694defb83f41c1572722ca) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20250627 13:56:38.087375 8239 maintenance_manager.cc:419] P e4c831df4da841929d13d6c783099b0e: Scheduling FlushMRSOp(644985e3086d4614b6598a727cb60c1d): perf score=0.033352
I20250627 13:56:38.137930 8130 maintenance_manager.cc:643] P e4c831df4da841929d13d6c783099b0e: FlushMRSOp(644985e3086d4614b6598a727cb60c1d) complete. Timing: real 0.050s user 0.035s sys 0.015s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":411,"dirs.run_cpu_time_us":260,"dirs.run_wall_time_us":1896,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":584,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":473,"threads_started":2,"wal-append.queue_time_us":290}
I20250627 13:56:38.138921 8239 maintenance_manager.cc:419] P e4c831df4da841929d13d6c783099b0e: Scheduling UndoDeltaBlockGCOp(644985e3086d4614b6598a727cb60c1d): 4926 bytes on disk
I20250627 13:56:38.139432 8130 maintenance_manager.cc:643] P e4c831df4da841929d13d6c783099b0e: UndoDeltaBlockGCOp(644985e3086d4614b6598a727cb60c1d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20250627 13:56:38.390949 8239 maintenance_manager.cc:419] P e4c831df4da841929d13d6c783099b0e: Scheduling FlushMRSOp(e3053b529a264e8697a1df9e17eeea81): perf score=0.033389
I20250627 13:56:38.422659 8130 maintenance_manager.cc:643] P e4c831df4da841929d13d6c783099b0e: FlushMRSOp(e3053b529a264e8697a1df9e17eeea81) complete. Timing: real 0.031s user 0.022s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1301,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":543,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":5888,"thread_start_us":207,"threads_started":1,"wal-append.queue_time_us":310}
I20250627 13:56:38.423870 8239 maintenance_manager.cc:419] P e4c831df4da841929d13d6c783099b0e: Scheduling UndoDeltaBlockGCOp(e3053b529a264e8697a1df9e17eeea81): 4924 bytes on disk
I20250627 13:56:38.424445 8130 maintenance_manager.cc:643] P e4c831df4da841929d13d6c783099b0e: UndoDeltaBlockGCOp(e3053b529a264e8697a1df9e17eeea81) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20250627 13:56:39.009994 7979 maintenance_manager.cc:419] P a41a89ae8ed743edb2b5cf4490d8c854: Scheduling FlushMRSOp(03640a5124d2405faf4a26d0e22e1ffc): perf score=0.033345
I20250627 13:56:39.038208 7870 maintenance_manager.cc:643] P a41a89ae8ed743edb2b5cf4490d8c854: FlushMRSOp(03640a5124d2405faf4a26d0e22e1ffc) complete. Timing: real 0.028s user 0.023s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":313,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1362,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":572,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":429,"threads_started":2,"wal-append.queue_time_us":328}
I20250627 13:56:39.039407 7979 maintenance_manager.cc:419] P a41a89ae8ed743edb2b5cf4490d8c854: Scheduling UndoDeltaBlockGCOp(03640a5124d2405faf4a26d0e22e1ffc): 4586 bytes on disk
I20250627 13:56:39.039978 7870 maintenance_manager.cc:643] P a41a89ae8ed743edb2b5cf4490d8c854: UndoDeltaBlockGCOp(03640a5124d2405faf4a26d0e22e1ffc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
W20250627 13:56:41.044281 7648 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet e2bd060a4a9e4765b45b0242ae56e489 with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER failed with leader 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101) due to error UNKNOWN_ERROR; will retry: Invalid argument: Cannot remove peer 2a1282ebe92a4c9a92eed3b13f69012a from the config because it is the leader. Force another leader to be elected to remove this peer. Consensus state: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:56:41.298611 9889 maintenance_manager.cc:419] P 956ff058e6d64579a1963aabdb6d466b: Scheduling FlushMRSOp(87726824b05a4b968386f57d3a7a4aad): perf score=0.033357
I20250627 13:56:41.336364 9779 maintenance_manager.cc:643] P 956ff058e6d64579a1963aabdb6d466b: FlushMRSOp(87726824b05a4b968386f57d3a7a4aad) complete. Timing: real 0.037s user 0.030s sys 0.005s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":401,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1437,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":751,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":1408,"thread_start_us":422,"threads_started":2,"wal-append.queue_time_us":287}
I20250627 13:56:41.337327 9889 maintenance_manager.cc:419] P 956ff058e6d64579a1963aabdb6d466b: Scheduling UndoDeltaBlockGCOp(87726824b05a4b968386f57d3a7a4aad): 4840 bytes on disk
I20250627 13:56:41.337869 9779 maintenance_manager.cc:643] P 956ff058e6d64579a1963aabdb6d466b: UndoDeltaBlockGCOp(87726824b05a4b968386f57d3a7a4aad) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
W20250627 13:57:41.072181 7648 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet e2bd060a4a9e4765b45b0242ae56e489 with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER failed with leader 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101) due to error UNKNOWN_ERROR; will retry: Invalid argument: Cannot remove peer 2a1282ebe92a4c9a92eed3b13f69012a from the config because it is the leader. Force another leader to be elected to remove this peer. Consensus state: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
W20250627 13:58:41.114339 7648 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet e2bd060a4a9e4765b45b0242ae56e489 with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER failed with leader 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101) due to error UNKNOWN_ERROR; will retry: Invalid argument: Cannot remove peer 2a1282ebe92a4c9a92eed3b13f69012a from the config because it is the leader. Force another leader to be elected to remove this peer. Consensus state: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
W20250627 13:59:41.142877 7648 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet e2bd060a4a9e4765b45b0242ae56e489 with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER failed with leader 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101) due to error UNKNOWN_ERROR; will retry: Invalid argument: Cannot remove peer 2a1282ebe92a4c9a92eed3b13f69012a from the config because it is the leader. Force another leader to be elected to remove this peer. Consensus state: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
I20250627 13:59:42.378104 7650 maintenance_manager.cc:419] P e7a954e8f6cf484a984a188e218b202d: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 14999 bytes on disk
I20250627 13:59:42.383123 7649 maintenance_manager.cc:643] P e7a954e8f6cf484a984a188e218b202d: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.004s sys 0.000s Metrics: {"dirs.queue_time_us":339,"dirs.run_cpu_time_us":99,"dirs.run_wall_time_us":98,"thread_start_us":470,"threads_started":2,"wal-append.queue_time_us":319}
W20250627 14:00:41.167353 7648 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet e2bd060a4a9e4765b45b0242ae56e489 with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER failed with leader 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101) due to error UNKNOWN_ERROR; will retry: Invalid argument: Cannot remove peer 2a1282ebe92a4c9a92eed3b13f69012a from the config because it is the leader. Force another leader to be elected to remove this peer. Consensus state: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
W20250627 14:01:41.205278 7648 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet e2bd060a4a9e4765b45b0242ae56e489 with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER failed with leader 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101) due to error UNKNOWN_ERROR; will retry: Invalid argument: Cannot remove peer 2a1282ebe92a4c9a92eed3b13f69012a from the config because it is the leader. Force another leader to be elected to remove this peer. Consensus state: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
W20250627 14:02:41.218389 7648 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet e2bd060a4a9e4765b45b0242ae56e489 with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER failed with leader 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101) due to error UNKNOWN_ERROR; will retry: Invalid argument: Cannot remove peer 2a1282ebe92a4c9a92eed3b13f69012a from the config because it is the leader. Force another leader to be elected to remove this peer. Consensus state: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
W20250627 14:03:41.238463 7648 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet e2bd060a4a9e4765b45b0242ae56e489 with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER failed with leader 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101) due to error UNKNOWN_ERROR; will retry: Invalid argument: Cannot remove peer 2a1282ebe92a4c9a92eed3b13f69012a from the config because it is the leader. Force another leader to be elected to remove this peer. Consensus state: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
W20250627 14:04:41.257416 7648 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet e2bd060a4a9e4765b45b0242ae56e489 with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER failed with leader 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101) due to error UNKNOWN_ERROR; will retry: Invalid argument: Cannot remove peer 2a1282ebe92a4c9a92eed3b13f69012a from the config because it is the leader. Force another leader to be elected to remove this peer. Consensus state: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
W20250627 14:05:41.288326 7648 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet e2bd060a4a9e4765b45b0242ae56e489 with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER failed with leader 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101) due to error UNKNOWN_ERROR; will retry: Invalid argument: Cannot remove peer 2a1282ebe92a4c9a92eed3b13f69012a from the config because it is the leader. Force another leader to be elected to remove this peer. Consensus state: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
W20250627 14:06:41.322990 7648 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet e2bd060a4a9e4765b45b0242ae56e489 with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER failed with leader 2a1282ebe92a4c9a92eed3b13f69012a (127.0.103.69:33101) due to error UNKNOWN_ERROR; will retry: Invalid argument: Cannot remove peer 2a1282ebe92a4c9a92eed3b13f69012a from the config because it is the leader. Force another leader to be elected to remove this peer. Consensus state: current_term: 2 leader_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9cf479bdcd0242738d2a77db4ad9f660" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38933 } attrs { replace: true } } peers { permanent_uuid: "2a1282ebe92a4c9a92eed3b13f69012a" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 33101 } attrs { promote: false } } peers { permanent_uuid: "e4c831df4da841929d13d6c783099b0e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46139 } attrs { promote: false } } peers { permanent_uuid: "83574c1bf8ea45adaff939158f93c096" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37467 } attrs { promote: false } } peers { permanent_uuid: "956ff058e6d64579a1963aabdb6d466b" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 40647 } attrs { promote: false } } }
************************ BEGIN STACKS **************************
[New LWP 416]
[New LWP 417]
[New LWP 7623]
[New LWP 7624]
[New LWP 7625]
[New LWP 7626]
[New LWP 7627]
[New LWP 8761]
[New LWP 8762]
[New LWP 8763]
[New LWP 8764]
[New LWP 10153]
[New LWP 10154]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f854e8f0d2d in __pthread_timedjoin_ex () from /lib/x86_64-linux-gnu/libpthread.so.0
Id Target Id Frame
* 1 Thread 0x7f8543b24200 (LWP 413) "rebalancer_tool" 0x00007f854e8f0d2d in __pthread_timedjoin_ex () from /lib/x86_64-linux-gnu/libpthread.so.0
2 Thread 0x7f853f6bd700 (LWP 416) "rebalancer_tool" 0x00007f854e8fa32a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
3 Thread 0x7f853d6c4700 (LWP 417) "kernel-watcher-" 0x00007f854e8f5fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
4 Thread 0x7f853b6c0700 (LWP 7623) "rpc reactor-762" 0x00007f85446bda47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
5 Thread 0x7f853bec1700 (LWP 7624) "rpc reactor-762" 0x00007f85446bda47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
6 Thread 0x7f853aebf700 (LWP 7625) "rpc reactor-762" 0x00007f85446bda47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
7 Thread 0x7f853cec3700 (LWP 7626) "rpc reactor-762" 0x00007f85446bda47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
8 Thread 0x7f853c6c2700 (LWP 7627) "rpc reactor-762" 0x00007f85446bda47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
9 Thread 0x7f853a6be700 (LWP 8761) "rpc reactor-876" 0x00007f85446bda47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
10 Thread 0x7f8539c76700 (LWP 8762) "rpc reactor-876" 0x00007f85446bda47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
11 Thread 0x7f8539467700 (LWP 8763) "rpc reactor-876" 0x00007f85446bda47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
12 Thread 0x7f8537c8a700 (LWP 8764) "rpc reactor-876" 0x00007f85446bda47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
13 Thread 0x7f8537479700 (LWP 10153) "rebalancer_tool" 0x00007f85446bda47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
14 Thread 0x7f8536477700 (LWP 10154) "rebalancer_tool" 0x00007f85446bda47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
Thread 14 (Thread 0x7f8536477700 (LWP 10154)):
#0 0x00007f85446bda47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f854636925d in epoll_poll (loop=0x61a000194a80, timeout=59.743000000000002) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f854636dba3 in ev_run (loop=0x61a000194a80, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f854a7ce6b1 in ev::loop_ref::run (this=<optimized out>, flags=888064) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f854848b5ab in kudu::(anonymous namespace)::ReadFdsFully (progname=..., fds=..., out=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/subprocess.cc:241
#5 0x00007f854848a72b in kudu::Subprocess::Call (argv=std::vector of length 12, capacity 16 = {...}, stdin_in=..., stdout_out=<optimized out>, stderr_out=<optimized out>, env_vars=std::map with 106996225575264 elements<error reading variable: Cannot access memory at address 0xccccfffdf7ade8ef>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/subprocess.cc:793
#6 0x00007f855e153212 in kudu::tools::RunKuduTool (args=..., out=<optimized out>, err=<optimized out>, in=..., env_vars=std::map with 0 elements) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/tool_test_util.cc:87
#7 0x00000000006375ab in kudu::tools::ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test::TestBody()::$_22::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1253
#8 0x000000000063728c in kudu::tools::ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test::TestBody()::$_12::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1272
#9 0x000000000063716e in _ZSt13__invoke_implIvZN4kudu5tools55ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test8TestBodyEvE4$_12JEET_St14__invoke_otherOT0_DpOT1_ (__f=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:60
#10 0x00000000006370ee in _ZSt8__invokeIZN4kudu5tools55ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test8TestBodyEvE4$_12JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS5_DpOS6_ (__fn=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
#11 0x00000000006370b3 in std::thread::_Invoker<std::tuple<kudu::tools::ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test::TestBody()::$_12> >::_M_invoke<0ul> (this=0x6030001522c8) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
#12 0x000000000063707e in std::thread::_Invoker<std::tuple<kudu::tools::ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test::TestBody()::$_12> >::operator()() (this=0x20, this@entry=0x6030001522c8) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
#13 0x0000000000636f69 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<kudu::tools::ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test::TestBody()::$_12> > >::_M_run() (this=<optimized out>) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
#14 0x00007f8544c626df in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#15 0x00007f854e8ef6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#16 0x00007f85446bd71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 13 (Thread 0x7f8537479700 (LWP 10153)):
#0 0x00007f85446bda47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f854636925d in epoll_poll (loop=0x61a000177c80, timeout=59.743000000000002) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f854636dba3 in ev_run (loop=0x61a000177c80, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f854a7ce6b1 in ev::loop_ref::run (this=<optimized out>, flags=758144) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f854848b5ab in kudu::(anonymous namespace)::ReadFdsFully (progname=..., fds=..., out=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/subprocess.cc:241
#5 0x00007f854848a72b in kudu::Subprocess::Call (argv=std::vector of length 12, capacity 16 = {...}, stdin_in=..., stdout_out=<optimized out>, stderr_out=<optimized out>, env_vars=std::map with 106996225823584 elements<error reading variable: Cannot access memory at address 0xccccfffdf7ade8ef>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/subprocess.cc:793
#6 0x00007f855e153212 in kudu::tools::RunKuduTool (args=..., out=<optimized out>, err=<optimized out>, in=..., env_vars=std::map with 0 elements) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/tool_test_util.cc:87
#7 0x00000000006375ab in kudu::tools::ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test::TestBody()::$_22::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1253
#8 0x000000000063728c in kudu::tools::ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test::TestBody()::$_12::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1272
#9 0x000000000063716e in _ZSt13__invoke_implIvZN4kudu5tools55ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test8TestBodyEvE4$_12JEET_St14__invoke_otherOT0_DpOT1_ (__f=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:60
#10 0x00000000006370ee in _ZSt8__invokeIZN4kudu5tools55ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test8TestBodyEvE4$_12JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS5_DpOS6_ (__fn=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
#11 0x00000000006370b3 in std::thread::_Invoker<std::tuple<kudu::tools::ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test::TestBody()::$_12> >::_M_invoke<0ul> (this=0x603000152268) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
#12 0x000000000063707e in std::thread::_Invoker<std::tuple<kudu::tools::ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test::TestBody()::$_12> >::operator()() (this=0x30, this@entry=0x603000152268) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
#13 0x0000000000636f69 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<kudu::tools::ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test::TestBody()::$_12> > >::_M_run() (this=<optimized out>) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
#14 0x00007f8544c626df in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#15 0x00007f854e8ef6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#16 0x00007f85446bd71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 12 (Thread 0x7f8537c8a700 (LWP 8764)):
#0 0x00007f85446bda47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f854636925d in epoll_poll (loop=0x61a00015c680, timeout=0.099220388168305362) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f854636dba3 in ev_run (loop=0x61a00015c680, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f854a7ce6b1 in ev::loop_ref::run (this=this@entry=0x61400013b2a0, flags=695424, flags@entry=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f854a7c6af6 in kudu::rpc::ReactorThread::RunThread (this=0x61400013b298) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510
#5 0x00007f854a7cbc1c in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197
#6 0x00007f854a7cba99 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007f855cda945d in std::function<void ()>::operator()() const (this=0x61100012ce58) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007f854849d4de in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:698
#9 0x00007f854e8ef6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f85446bd71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 11 (Thread 0x7f8539467700 (LWP 8763)):
#0 0x00007f85446bda47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f854636925d in epoll_poll (loop=0x61a00015c080, timeout=0.099477358168542196) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f854636dba3 in ev_run (loop=0x61a00015c080, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f854a7ce6b1 in ev::loop_ref::run (this=this@entry=0x61400013b0a0, flags=694528, flags@entry=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f854a7c6af6 in kudu::rpc::ReactorThread::RunThread (this=0x61400013b098) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510
#5 0x00007f854a7cbc1c in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197
#6 0x00007f854a7cba99 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007f855cda945d in std::function<void ()>::operator()() const (this=0x61100012d0d8) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007f854849d4de in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:698
#9 0x00007f854e8ef6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f85446bd71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 10 (Thread 0x7f8539c76700 (LWP 8762)):
#0 0x00007f85446bda47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f854636925d in epoll_poll (loop=0x61a00015ba80, timeout=0.099201718168387742) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f854636dba3 in ev_run (loop=0x61a00015ba80, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f854a7ce6b1 in ev::loop_ref::run (this=this@entry=0x61400013aea0, flags=693632, flags@entry=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f854a7c6af6 in kudu::rpc::ReactorThread::RunThread (this=0x61400013ae98) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510
#5 0x00007f854a7cbc1c in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197
#6 0x00007f854a7cba99 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007f855cda945d in std::function<void ()>::operator()() const (this=0x61100012d218) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007f854849d4de in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:698
#9 0x00007f854e8ef6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f85446bd71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 9 (Thread 0x7f853a6be700 (LWP 8761)):
#0 0x00007f85446bda47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f854636925d in epoll_poll (loop=0x61a00015b480, timeout=0.099138804168433126) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f854636dba3 in ev_run (loop=0x61a00015b480, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f854a7ce6b1 in ev::loop_ref::run (this=this@entry=0x61400013aca0, flags=692736, flags@entry=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f854a7c6af6 in kudu::rpc::ReactorThread::RunThread (this=0x61400013ac98) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510
#5 0x00007f854a7cbc1c in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197
#6 0x00007f854a7cba99 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007f855cda945d in std::function<void ()>::operator()() const (this=0x61100012d358) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007f854849d4de in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:698
#9 0x00007f854e8ef6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f85446bd71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 8 (Thread 0x7f853c6c2700 (LWP 7627)):
#0 0x00007f85446bda47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f854636925d in epoll_poll (loop=0x61a000157280, timeout=0.099834594168214608) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f854636dba3 in ev_run (loop=0x61a000157280, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f854a7ce6b1 in ev::loop_ref::run (this=this@entry=0x614000138ea0, flags=690944, flags@entry=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f854a7c6af6 in kudu::rpc::ReactorThread::RunThread (this=0x614000138e98) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510
#5 0x00007f854a7cbc1c in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197
#6 0x00007f854a7cba99 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007f855cda945d in std::function<void ()>::operator()() const (this=0x611000146718) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007f854849d4de in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:698
#9 0x00007f854e8ef6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f85446bd71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 7 (Thread 0x7f853cec3700 (LWP 7626)):
#0 0x00007f85446bda47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f854636925d in epoll_poll (loop=0x61a000156c80, timeout=0.099794911168146427) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f854636dba3 in ev_run (loop=0x61a000156c80, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f854a7ce6b1 in ev::loop_ref::run (this=this@entry=0x6140001388a0, flags=690048, flags@entry=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f854a7c6af6 in kudu::rpc::ReactorThread::RunThread (this=0x614000138898) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510
#5 0x00007f854a7cbc1c in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197
#6 0x00007f854a7cba99 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007f855cda945d in std::function<void ()>::operator()() const (this=0x611000146ad8) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007f854849d4de in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:698
#9 0x00007f854e8ef6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f85446bd71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 6 (Thread 0x7f853aebf700 (LWP 7625)):
#0 0x00007f85446bda47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f854636925d in epoll_poll (loop=0x61a000156680, timeout=0.099769843168360239) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f854636dba3 in ev_run (loop=0x61a000156680, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f854a7ce6b1 in ev::loop_ref::run (this=this@entry=0x6140001386a0, flags=689152, flags@entry=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f854a7c6af6 in kudu::rpc::ReactorThread::RunThread (this=0x614000138698) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510
#5 0x00007f854a7cbc1c in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197
#6 0x00007f854a7cba99 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007f855cda945d in std::function<void ()>::operator()() const (this=0x611000146c18) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007f854849d4de in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:698
#9 0x00007f854e8ef6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f85446bd71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 5 (Thread 0x7f853bec1700 (LWP 7624)):
#0 0x00007f85446bda47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f854636925d in epoll_poll (loop=0x61a000156080, timeout=0.09974180516837805) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f854636dba3 in ev_run (loop=0x61a000156080, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f854a7ce6b1 in ev::loop_ref::run (this=this@entry=0x6140001384a0, flags=688256, flags@entry=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f854a7c6af6 in kudu::rpc::ReactorThread::RunThread (this=0x614000138498) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510
#5 0x00007f854a7cbc1c in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197
#6 0x00007f854a7cba99 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007f855cda945d in std::function<void ()>::operator()() const (this=0x611000146fd8) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007f854849d4de in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:698
#9 0x00007f854e8ef6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f85446bd71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 4 (Thread 0x7f853b6c0700 (LWP 7623)):
#0 0x00007f85446bda47 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f854636925d in epoll_poll (loop=0x61a000155a80, timeout=0.099687783168292299) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.33/ev_epoll.c:155
#2 0x00007f854636dba3 in ev_run (loop=0x61a000155a80, flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/libev-4.33/ev.c:4157
#3 0x00007f854a7ce6b1 in ev::loop_ref::run (this=this@entry=0x6140001382a0, flags=687360, flags@entry=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/ev++.h:211
#4 0x00007f854a7c6af6 in kudu::rpc::ReactorThread::RunThread (this=0x614000138298) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:510
#5 0x00007f854a7cbc1c in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/reactor.cc:197
#6 0x00007f854a7cba99 in std::_Function_handler<void (), kudu::rpc::ReactorThread::Init()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007f855cda945d in std::function<void ()>::operator()() const (this=0x611000146e98) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007f854849d4de in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:698
#9 0x00007f854e8ef6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f85446bd71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 3 (Thread 0x7f853d6c4700 (LWP 417)):
#0 0x00007f854e8f5fb9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f854820306e in kudu::ConditionVariable::WaitUntil (this=0x612000020f98, until=...) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/condition_variable.cc:87
#2 0x00000000006d4f0c in kudu::CountDownLatch::WaitUntil (this=<optimized out>, when=...) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/countdown_latch.h:89
#3 0x00000000006d4d5a in kudu::CountDownLatch::WaitFor (this=0x612000020fc0, delta=...) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/countdown_latch.h:99
#4 0x00007f854833b3df in kudu::KernelStackWatchdog::RunThread (this=0x612000020ec0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/kernel_stack_watchdog.cc:134
#5 0x00007f854833c3ac in kudu::KernelStackWatchdog::KernelStackWatchdog()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/kernel_stack_watchdog.cc:71
#6 0x00007f854833c229 in std::_Function_handler<void (), kudu::KernelStackWatchdog::KernelStackWatchdog()::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
#7 0x00007f855cda945d in std::function<void ()>::operator()() const (this=0x611000060c18) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#8 0x00007f854849d4de in kudu::Thread::SuperviseThread (arg=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:698
#9 0x00007f854e8ef6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007f85446bd71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 2 (Thread 0x7f853f6bd700 (LWP 416)):
#0 0x00007f854e8fa32a in waitpid () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000005454bf in __interceptor_waitpid (pid=<optimized out>, status=0x7f853f6bb710, options=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:2434
#2 0x00007f8548487005 in kudu::Subprocess::DoWait (this=<optimized out>, wait_status=<optimized out>, mode=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/subprocess.cc:849
#3 0x00007f8548486c85 in kudu::Subprocess::Wait (this=0x7f853f6bb710, wait_status=0x0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/subprocess.cc:561
#4 0x00007f85483154be in kudu::PstackWatcher::RunStackDump (argv=...) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/pstack_watcher.cc:260
#5 0x00007f8548314d44 in kudu::PstackWatcher::RunGdbStackDump (pid=<optimized out>, flags=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/pstack_watcher.cc:218
#6 0x00007f85483141d9 in kudu::PstackWatcher::DumpPidStacks (pid=<optimized out>, flags=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/pstack_watcher.cc:169
#7 0x00007f8548312567 in kudu::PstackWatcher::DumpStacks (flags=0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/pstack_watcher.cc:162
#8 0x00000000006ff55b in kudu::CreateAndStartTimeoutThread()::$_0::operator()() const (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:60
#9 0x00000000006ff3f3 in _ZSt13__invoke_implIvZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0JEET_St14__invoke_otherOT0_DpOT1_ (__f=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:60
#10 0x00000000006ff373 in _ZSt8__invokeIZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS3_DpOS4_ (__fn=...) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
#11 0x00000000006ff349 in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::$_0> >::_M_invoke<0ul> (this=this@entry=0x602000013098) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
#12 0x00000000006ff309 in std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::$_0> >::operator()() (this=this@entry=0x602000013098) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
#13 0x00000000006ff20b in std::thread::_State_impl<std::thread::_Invoker<std::tuple<kudu::CreateAndStartTimeoutThread()::$_0> > >::_M_run() (this=<optimized out>) at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
#14 0x00007f8544c626df in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#15 0x00007f854e8ef6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#16 0x00007f85446bd71f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 1 (Thread 0x7f8543b24200 (LWP 413)):
#0 0x00007f854e8f0d2d in __pthread_timedjoin_ex () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f8544c62933 in std::thread::join() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2 0x0000000000607019 in kudu::tools::ConcurrentRebalancersTest_TwoConcurrentRebalancers_Test::TestBody (this=0x619000b72580) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1279
#3 0x00007f855c69a0ed in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (location=0x7f855c69b82b "the test body", method=<optimized out>, object=0x619000b72580) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599
#4 testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=object@entry=0x619000b72580, method=<optimized out>, location=location@entry=0x7f855c69b82b "the test body") at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#5 0x00007f855c68ebdb in testing::Test::Run (this=0x619000b72580) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2674
#6 0x00007f855c68ed9d in testing::TestInfo::Run (this=0x612000004540) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2853
#7 0x00007f855c68f377 in testing::TestSuite::Run (this=0x6120000043c0) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012
#8 0x00007f855c68f77c in testing::internal::UnitTestImpl::RunAllTests (this=<optimized out>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5870
#9 0x00007f855c69a60d in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (location=0x7f855c69d3e8 "auxiliary test code (environments or event listeners)", method=<optimized out>, object=0x616000005a80) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599
#10 testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x616000005a80, method=<optimized out>, location=location@entry=0x7f855c69d3e8 "auxiliary test code (environments or event listeners)") at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635
#11 0x00007f855c68ee63 in testing::UnitTest::Run (this=0x7f855c8ae1e0 <testing::UnitTest::GetInstance()::instance>) at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5444
#12 0x00000000007001f2 in RUN_ALL_TESTS () at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/uninstrumented/include/gtest/gtest.h:2293
#13 0x00000000006fe7f9 in main (argc=<optimized out>, argv=0x7ffcdd1a0cd8) at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:109
************************* END STACKS ***************************
F20250627 14:07:13.343544 416 test_main.cc:63] Maximum unit test time exceeded (870 sec)
*** Check failure stack trace: ***
*** Aborted at 1751033233 (unix time) try "date -d @1751033233" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e80000019d) received by PID 413 (TID 0x7f853f6bd700) from PID 413; stack trace: ***
@ 0x7f854e8fa980 (unknown) at ??:0
@ 0x7f85445dafb7 gsignal at ??:0
@ 0x7f85445dc921 abort at ??:0
@ 0x7f8546e34dcd google::LogMessage::Fail() at ??:0
@ 0x7f8546e38b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f8546e347cc google::LogMessage::Flush() at ??:0
@ 0x7f8546e35f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x6ff6f9 kudu::CreateAndStartTimeoutThread()::$_0::operator()() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:63
@ 0x6ff3f3 std::__invoke_impl<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:60
@ 0x6ff373 std::__invoke<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
@ 0x6ff349 _ZNSt6thread8_InvokerISt5tupleIJZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0EEE9_M_invokeIJLm0EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
@ 0x6ff309 std::thread::_Invoker<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
@ 0x6ff20b std::thread::_State_impl<>::_M_run() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
@ 0x7f8544c626df (unknown) at ??:0
@ 0x7f854e8ef6db start_thread at ??:0
@ 0x7f85446bd71f clone at ??:0